[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:19:14 dist-test-slave-777x krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:19:14 dist-test-slave-777x krb5kdc[399](info): set up 2 sockets
Sep 24 00:19:14 dist-test-slave-777x krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 00:19:15.429402 392 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:19:15 dist-test-slave-777x krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:19:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:19:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:19:15.559247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39695
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:15.640576 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:15.640839 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:15.640877 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:15.642114 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:19:15.642191 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:15.642215 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:15.642235 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:19:15.642266 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:19:15.643817 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:15.644110 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:15.644418 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:15.650238 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:15.650233 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:15.650445 412 server_base.cc:1034] running on GCE node
W20260924 00:19:15.650252 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:15.650792 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:15.651118 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:15.652275 412 hybrid_clock.cc:648] HybridClock initialized: now 1790209155652198 us; error 88 us; skew 500 ppm
Sep 24 00:19:15 dist-test-slave-777x krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:15.655226 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:19:15.655761 412 webserver.cc:460] Webserver started at http://127.0.98.62:45821/ using document root <none> and password file <none>
I20260924 00:19:15.656056 412 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:15.656108 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:15.656390 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:15.658031 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "1feb9ac5625e4c94a9a1333a84270474"
format_stamp: "Formatted at 2026-09-24 00:19:15 on dist-test-slave-777x"
server_key: "2caec6d429bcea6b79697a79e573ca99"
server_key_iv: "2425422dc3679d78110a368bde1d6371"
server_key_version: "encryptionkey@0"
I20260924 00:19:15.658456 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "1feb9ac5625e4c94a9a1333a84270474"
format_stamp: "Formatted at 2026-09-24 00:19:15 on dist-test-slave-777x"
server_key: "2caec6d429bcea6b79697a79e573ca99"
server_key_iv: "2425422dc3679d78110a368bde1d6371"
server_key_version: "encryptionkey@0"
I20260924 00:19:15.660120 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:19:15.661105 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:15.661352 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:15.661466 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "1feb9ac5625e4c94a9a1333a84270474"
format_stamp: "Formatted at 2026-09-24 00:19:15 on dist-test-slave-777x"
server_key: "2caec6d429bcea6b79697a79e573ca99"
server_key_iv: "2425422dc3679d78110a368bde1d6371"
server_key_version: "encryptionkey@0"
I20260924 00:19:15.661562 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:15.692392 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:15.694607 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:15.694803 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:15.699380 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39695
I20260924 00:19:15.699421 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39695 every 8 connection(s)
I20260924 00:19:15.699828 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:19:15.700865 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:15.703109 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474: Bootstrap starting.
I20260924 00:19:15.703670 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:15.703897 480 log.cc:824] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:15.704571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 412
I20260924 00:19:15.704653 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:19:15.704797 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474: No bootstrap required, opened a new log
I20260924 00:19:15.705302 392 external_mini_cluster.cc:1428] Setting key 0684ecfe0396c04153435053cf59e0b3
I20260924 00:19:15.707072 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1feb9ac5625e4c94a9a1333a84270474" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } }
I20260924 00:19:15.707276 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:15.707321 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1feb9ac5625e4c94a9a1333a84270474, State: Initialized, Role: FOLLOWER
I20260924 00:19:15.707453 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1feb9ac5625e4c94a9a1333a84270474" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } }
I20260924 00:19:15.707544 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:19:15.707578 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:19:15.707628 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:15.708899 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1feb9ac5625e4c94a9a1333a84270474" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } }
I20260924 00:19:15.709023 480 leader_election.cc:302] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1feb9ac5625e4c94a9a1333a84270474; no voters:
I20260924 00:19:15.709224 480 leader_election.cc:288] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:19:15.709288 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:15.709460 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474 [term 1 LEADER]: Becoming Leader. State: Replica: 1feb9ac5625e4c94a9a1333a84270474, State: Running, Role: LEADER
I20260924 00:19:15.709559 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:19:15.709591 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1feb9ac5625e4c94a9a1333a84270474" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } }
I20260924 00:19:15.710084 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1feb9ac5625e4c94a9a1333a84270474" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1feb9ac5625e4c94a9a1333a84270474" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } }
I20260924 00:19:15.710203 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:15.710103 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1feb9ac5625e4c94a9a1333a84270474. Latest consensus state: current_term: 1 leader_uuid: "1feb9ac5625e4c94a9a1333a84270474" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1feb9ac5625e4c94a9a1333a84270474" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39695 } } }
I20260924 00:19:15.710427 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:15.710423 493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:19:15.711131 493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:19:15.712738 493 catalog_manager.cc:1347] Generated new cluster ID: e33182508e1449b3a6810374216381c8
I20260924 00:19:15.712796 493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:19:15 dist-test-slave-777x krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:15.725090 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:15.706216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46200 (local address 127.0.98.62:39695)
0924 00:19:15.706383 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:15.706392 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:15.706652 (+ 260us) server_negotiation.cc:407] Connection header received
0924 00:19:15.709473 (+ 2821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:15.709481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:15.709552 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:15.712066 (+ 2514us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:15.712188 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:15.713658 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:15.715238 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:15.715900 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:15.716020 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:15.720759 (+ 4739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:15.720765 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:15.720765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:15.720780 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:15.722943 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:15.723472 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:15.723475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:15.723476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:15.723586 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:15.723888 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:15.723888 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:15.723889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:15.724240 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:15.724393 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:15.724697 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 00:19:15.724841 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260924 00:19:15.730222 493 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:19:15.730842 493 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:19:15.735185 493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1feb9ac5625e4c94a9a1333a84270474: Generated new TSK 0
I20260924 00:19:15.735376 493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:19:15.768573 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39695
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:15.854099 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:15.854369 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:15.854396 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:15.855592 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:15.855641 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:15.855695 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:19:15.857245 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39695
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:15.857508 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:15.857764 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:15.862880 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:15.862864 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:15.862864 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:15.863031 506 server_base.cc:1034] running on GCE node
I20260924 00:19:15.863351 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:15.863617 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:15.864817 506 hybrid_clock.cc:648] HybridClock initialized: now 1790209155864795 us; error 30 us; skew 500 ppm
Sep 24 00:19:15 dist-test-slave-777x krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:15.867812 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:19:15.868374 506 webserver.cc:460] Webserver started at http://127.0.98.1:45503/ using document root <none> and password file <none>
I20260924 00:19:15.868620 506 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:15.868695 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:15.868858 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:15.870082 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "eee3ba08d11e4b30816a4db5efc898f2"
format_stamp: "Formatted at 2026-09-24 00:19:15 on dist-test-slave-777x"
server_key: "2c16cb59c9eca923f3f52a4a744e54d3"
server_key_iv: "c18c2f3edff0d008d9b432e2ed727149"
server_key_version: "encryptionkey@0"
I20260924 00:19:15.870508 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "eee3ba08d11e4b30816a4db5efc898f2"
format_stamp: "Formatted at 2026-09-24 00:19:15 on dist-test-slave-777x"
server_key: "2c16cb59c9eca923f3f52a4a744e54d3"
server_key_iv: "c18c2f3edff0d008d9b432e2ed727149"
server_key_version: "encryptionkey@0"
I20260924 00:19:15.872146 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:19:15.873080 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:15.873273 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:15.873414 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "eee3ba08d11e4b30816a4db5efc898f2"
format_stamp: "Formatted at 2026-09-24 00:19:15 on dist-test-slave-777x"
server_key: "2c16cb59c9eca923f3f52a4a744e54d3"
server_key_iv: "c18c2f3edff0d008d9b432e2ed727149"
server_key_version: "encryptionkey@0"
I20260924 00:19:15.873492 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:15.883052 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:15.885176 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:15.885357 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:15.885599 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:15.885905 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:15.885959 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:15.885993 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:15.886021 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:15.891759 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36623
I20260924 00:19:15.891811 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36623 every 8 connection(s)
I20260924 00:19:15.892153 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:19:15.893738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 506
I20260924 00:19:15.893872 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:19:15.894017 392 external_mini_cluster.cc:1428] Setting key 063ce173e3c68309d9df00605e647ef9
Sep 24 00:19:15 dist-test-slave-777x krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 00:19:15.903759 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:15.892939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44799 (local address 127.0.98.62:39695)
0924 00:19:15.893048 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:15.893050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:15.893168 (+ 118us) server_negotiation.cc:407] Connection header received
0924 00:19:15.893681 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:15.893691 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:15.893705 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:15.893777 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:15.894978 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:15.895780 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:15.897446 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:15.897665 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:15.897731 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:15.900506 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:15.900507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:15.900508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:15.900516 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:15.902158 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:15.902693 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:15.902695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:15.902695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:15.902761 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:15.903079 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:15.903080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:15.903080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:15.903254 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:15.903331 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:15.903603 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 00:19:15.903686 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 00:19:15.903935 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:15.892882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39695 (local address 127.0.98.1:44799)
0924 00:19:15.893051 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:15.893088 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:15.893583 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:15.893849 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:15.893872 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:15.894598 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:15.894881 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:15.894887 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:15.895884 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:15.895885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:15.897347 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:15.897350 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:15.897757 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:15.897759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:15.897905 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:15.898571 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:15.898583 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:15.900376 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:15.902323 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:15.902331 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:15.902334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:15.902588 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:15.902845 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:15.902846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:15.902846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:15.902998 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:15.903336 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:15.903342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:15.903474 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 00:19:15.903574 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 00:19:15.903707 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
I20260924 00:19:15.904588 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39695
I20260924 00:19:15.904716 635 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 00:19:15.904971 635 heartbeater.cc:511] Master 127.0.98.62:39695 requested a full tablet report, sending...
I20260924 00:19:15.905607 444 ts_manager.cc:194] Registered new tserver with Master: eee3ba08d11e4b30816a4db5efc898f2 (127.0.98.1:36623)
I20260924 00:19:15.906625 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:19:15.922657 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39695
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:16.003897 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:16.004104 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:16.004127 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:16.005240 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:16.005283 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:16.005311 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:19:16.006651 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39695
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:16.006907 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:16.007094 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:16.012079 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:16.012326 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:16.012357 640 server_base.cc:1034] running on GCE node
W20260924 00:19:16.012082 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:16.012655 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:16.012912 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:16.014055 640 hybrid_clock.cc:648] HybridClock initialized: now 1790209156014026 us; error 50 us; skew 500 ppm
Sep 24 00:19:16 dist-test-slave-777x krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:16.016738 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:19:16.017254 640 webserver.cc:460] Webserver started at http://127.0.98.2:41697/ using document root <none> and password file <none>
I20260924 00:19:16.017513 640 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:16.017560 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:16.017732 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:16.018821 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "f6acd1114f6f446da82968d72c118523"
format_stamp: "Formatted at 2026-09-24 00:19:16 on dist-test-slave-777x"
server_key: "979d9d1216b9cbbf7be4da4e8ad4f826"
server_key_iv: "1f44e9a287db4e84ba8585949c9ee81a"
server_key_version: "encryptionkey@0"
I20260924 00:19:16.019187 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f6acd1114f6f446da82968d72c118523"
format_stamp: "Formatted at 2026-09-24 00:19:16 on dist-test-slave-777x"
server_key: "979d9d1216b9cbbf7be4da4e8ad4f826"
server_key_iv: "1f44e9a287db4e84ba8585949c9ee81a"
server_key_version: "encryptionkey@0"
I20260924 00:19:16.020610 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:19:16.021566 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:16.021790 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:16.021888 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "f6acd1114f6f446da82968d72c118523"
format_stamp: "Formatted at 2026-09-24 00:19:16 on dist-test-slave-777x"
server_key: "979d9d1216b9cbbf7be4da4e8ad4f826"
server_key_iv: "1f44e9a287db4e84ba8585949c9ee81a"
server_key_version: "encryptionkey@0"
I20260924 00:19:16.021960 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:16.029561 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:16.031404 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:16.031558 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:16.031770 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:16.032060 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:16.032092 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:16.032114 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:16.032199 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:16.038770 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37757
I20260924 00:19:16.038849 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37757 every 8 connection(s)
I20260924 00:19:16.039157 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 00:19:16 dist-test-slave-777x krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:16.047509 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 640
I20260924 00:19:16.047614 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:19:16.047787 392 external_mini_cluster.cc:1428] Setting key bdb7b7383c93e19551cef064a0fed20c
I20260924 00:19:16.049081 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:16.039984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51133 (local address 127.0.98.62:39695)
0924 00:19:16.040028 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:16.040032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:16.040289 (+ 257us) server_negotiation.cc:407] Connection header received
0924 00:19:16.040829 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:16.040830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:16.040843 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:16.040909 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:16.041872 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:16.042377 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:16.043407 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:16.043605 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:16.043691 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:16.045923 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:16.045924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:16.045924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:16.045931 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:16.047441 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:16.047995 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:16.047997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:16.047997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:16.048064 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:16.048358 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:16.048359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:16.048359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:16.048522 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:16.048593 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:16.048931 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 00:19:16.049018 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 00:19:16.049379 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:16.039954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39695 (local address 127.0.98.2:51133)
0924 00:19:16.040120 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:16.040198 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:16.040719 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:16.040990 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:16.040995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:16.041566 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:16.041769 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:16.041774 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:16.042470 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:16.042472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:16.043313 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:16.043316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:16.043795 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:16.043797 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:16.043900 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:16.044431 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:16.044440 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:16.045834 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:16.047528 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:16.047534 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:16.047537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:16.047915 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:16.048132 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:16.048133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:16.048133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:16.048276 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:16.048599 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:16.048606 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:16.048781 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 00:19:16.048891 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 00:19:16.049084 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260924 00:19:16.049928 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39695
I20260924 00:19:16.050046 769 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:16.050251 769 heartbeater.cc:511] Master 127.0.98.62:39695 requested a full tablet report, sending...
I20260924 00:19:16.050665 444 ts_manager.cc:194] Registered new tserver with Master: f6acd1114f6f446da82968d72c118523 (127.0.98.2:37757)
I20260924 00:19:16.051199 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51133
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:19:16.075778 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39695
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:16.159411 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:16.159636 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:16.159660 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:16.160869 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:16.160914 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:16.160941 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:19:16.162423 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39695
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:16.162616 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:16.162799 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:16.168156 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:16.168161 774 server_base.cc:1034] running on GCE node
W20260924 00:19:16.168133 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:16.168133 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:16.168670 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:16.168927 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:16.170091 774 hybrid_clock.cc:648] HybridClock initialized: now 1790209156170091 us; error 45 us; skew 500 ppm
Sep 24 00:19:16 dist-test-slave-777x krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:16.172709 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:19:16.173199 774 webserver.cc:460] Webserver started at http://127.0.98.3:34537/ using document root <none> and password file <none>
I20260924 00:19:16.173449 774 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:16.173498 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:16.173635 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:16.174726 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "c454d5ce4efe49da8d239ec64bda03f1"
format_stamp: "Formatted at 2026-09-24 00:19:16 on dist-test-slave-777x"
server_key: "87c35f3d5156b45c18ef328953624c5b"
server_key_iv: "6e331fc433611c3463fa4656de680fa5"
server_key_version: "encryptionkey@0"
I20260924 00:19:16.175107 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c454d5ce4efe49da8d239ec64bda03f1"
format_stamp: "Formatted at 2026-09-24 00:19:16 on dist-test-slave-777x"
server_key: "87c35f3d5156b45c18ef328953624c5b"
server_key_iv: "6e331fc433611c3463fa4656de680fa5"
server_key_version: "encryptionkey@0"
I20260924 00:19:16.176743 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:19:16.177546 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:16.177726 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:19:16.177851 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "c454d5ce4efe49da8d239ec64bda03f1"
format_stamp: "Formatted at 2026-09-24 00:19:16 on dist-test-slave-777x"
server_key: "87c35f3d5156b45c18ef328953624c5b"
server_key_iv: "6e331fc433611c3463fa4656de680fa5"
server_key_version: "encryptionkey@0"
I20260924 00:19:16.177932 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:16.187446 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:16.189916 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:16.190088 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:16.190806 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:16.191155 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:16.191196 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:16.191221 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:16.191272 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:16.199204 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45465
I20260924 00:19:16.199234 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45465 every 8 connection(s)
I20260924 00:19:16.199746 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:19:16.200827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 774
I20260924 00:19:16.200946 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:19:16.201119 392 external_mini_cluster.cc:1428] Setting key ade975177b7c9e7632c518a379486671
Sep 24 00:19:16 dist-test-slave-777x krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:16.209654 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:16.200667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58001 (local address 127.0.98.62:39695)
0924 00:19:16.200727 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:16.200729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:16.200985 (+ 256us) server_negotiation.cc:407] Connection header received
0924 00:19:16.201577 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:16.201578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:16.201592 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:16.201658 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:16.202634 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:16.203133 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:16.204218 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:16.204417 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:16.204505 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:16.206818 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:16.206819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:16.206820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:16.206828 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:16.208081 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:16.208604 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:16.208606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:16.208607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:16.208664 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:16.208930 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:16.208931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:16.208932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:16.209053 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:16.209131 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:16.209472 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 00:19:16.209554 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 00:19:16.209879 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:16.200611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39695 (local address 127.0.98.3:58001)
0924 00:19:16.200841 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:16.200881 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:16.201448 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:16.201732 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:16.201737 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:16.202326 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:16.202529 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:16.202534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:16.203244 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:16.203245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:16.204099 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:16.204101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:16.204512 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:16.204514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:16.204653 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:16.205244 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:16.205253 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:16.206694 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:16.208203 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:16.208210 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:16.208212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:16.208527 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:16.208735 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:16.208735 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:16.208736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:16.208860 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:16.209121 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:16.209128 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:16.209328 (+ 200us) client_negotiation.cc:769] Sending connection context
0924 00:19:16.209454 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 00:19:16.209621 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
I20260924 00:19:16.210453 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39695
I20260924 00:19:16.210565 903 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:16.210773 903 heartbeater.cc:511] Master 127.0.98.62:39695 requested a full tablet report, sending...
I20260924 00:19:16.211261 444 ts_manager.cc:194] Registered new tserver with Master: c454d5ce4efe49da8d239ec64bda03f1 (127.0.98.3:45465)
I20260924 00:19:16.211740 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58001
I20260924 00:19:16.213042 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 00:19:16 dist-test-slave-777x krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209156, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 00:19:16 dist-test-slave-777x krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209156, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:19:16.236116 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:16.228320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56960 (local address 127.0.98.1:36623)
0924 00:19:16.228492 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:16.228497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:16.228516 (+ 19us) server_negotiation.cc:407] Connection header received
0924 00:19:16.228544 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:16.228546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:16.228560 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:16.228634 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:16.229171 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:16.229949 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:16.230688 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:16.231020 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:16.231081 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:16.233389 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:16.233397 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:16.233397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:16.233404 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:16.234870 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:16.235188 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:16.235191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:16.235191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:16.235387 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:16.235584 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:16.235585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:16.235585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:16.235764 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:16.235900 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:16.235971 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 00:19:16.236035 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":104,"threads_started":1}
W20260924 00:19:16.236567 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56960
Sep 24 00:19:16 dist-test-slave-777x krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 00:19:16 dist-test-slave-777x krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:19:16.259044 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:16.251898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56968 (local address 127.0.98.1:36623)
0924 00:19:16.251968 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:16.251969 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:16.251995 (+ 26us) server_negotiation.cc:407] Connection header received
0924 00:19:16.252077 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:16.252077 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:16.252090 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:16.252141 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:16.252789 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:16.253270 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:16.253977 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:16.254175 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:16.254229 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:16.256574 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:16.256575 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:16.256575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:16.256584 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:16.257866 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:16.258262 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:16.258264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:16.258264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:16.258350 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:16.258558 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:16.258559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:16.258559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:16.258666 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:16.258764 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:16.258909 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 00:19:16.258977 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 00:19:16.259663 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 506
I20260924 00:19:16.264544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 640
I20260924 00:19:16.270318 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 774
I20260924 00:19:16.275338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 412
2026-09-24T00:19:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1965 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:19:16 dist-test-slave-777x krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:19:16 dist-test-slave-777x krb5kdc[917](info): set up 2 sockets
Sep 24 00:19:16 dist-test-slave-777x krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260924 00:19:17.347718 392 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:19:17 dist-test-slave-777x krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:19:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:19:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:19:17.472741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33819
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33819
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:17.554092 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:17.554309 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:17.554332 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:17.555552 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:19:17.555595 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:17.555610 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:17.555622 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:19:17.555634 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:19:17.557222 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:17.557487 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:17.557714 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:17.562561 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:17.562561 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:17.562602 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:17.562901 930 server_base.cc:1034] running on GCE node
I20260924 00:19:17.563138 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:17.563410 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:17.564589 930 hybrid_clock.cc:648] HybridClock initialized: now 1790209157564559 us; error 65 us; skew 500 ppm
Sep 24 00:19:17 dist-test-slave-777x krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:17.567067 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:19:17.567584 930 webserver.cc:460] Webserver started at http://127.0.98.62:44207/ using document root <none> and password file <none>
I20260924 00:19:17.567826 930 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:17.567894 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:17.568044 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:17.569205 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "690196df77374d2682847acb2c6d0fd0"
format_stamp: "Formatted at 2026-09-24 00:19:17 on dist-test-slave-777x"
server_key: "0db3d0c0ac5567ed1e75e66bf50f4adb"
server_key_iv: "ed54f32b5403f38563f1da8e88fb0412"
server_key_version: "encryptionkey@0"
I20260924 00:19:17.569618 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "690196df77374d2682847acb2c6d0fd0"
format_stamp: "Formatted at 2026-09-24 00:19:17 on dist-test-slave-777x"
server_key: "0db3d0c0ac5567ed1e75e66bf50f4adb"
server_key_iv: "ed54f32b5403f38563f1da8e88fb0412"
server_key_version: "encryptionkey@0"
I20260924 00:19:17.571043 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:19:17.571820 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:17.572029 930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:17.572129 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "690196df77374d2682847acb2c6d0fd0"
format_stamp: "Formatted at 2026-09-24 00:19:17 on dist-test-slave-777x"
server_key: "0db3d0c0ac5567ed1e75e66bf50f4adb"
server_key_iv: "ed54f32b5403f38563f1da8e88fb0412"
server_key_version: "encryptionkey@0"
I20260924 00:19:17.572242 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:17.592259 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:17.594894 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:17.595103 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:17.599560 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33819
I20260924 00:19:17.599611 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33819 every 8 connection(s)
I20260924 00:19:17.600014 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:19:17.600950 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:17.603190 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0: Bootstrap starting.
I20260924 00:19:17.603753 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:17.603997 998 log.cc:824] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:17.604691 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0: No bootstrap required, opened a new log
I20260924 00:19:17.606282 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690196df77374d2682847acb2c6d0fd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33819 } }
I20260924 00:19:17.606468 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:17.606494 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 690196df77374d2682847acb2c6d0fd0, State: Initialized, Role: FOLLOWER
I20260924 00:19:17.606604 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690196df77374d2682847acb2c6d0fd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33819 } }
I20260924 00:19:17.606701 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:19:17.606738 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:19:17.606791 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:17.607399 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690196df77374d2682847acb2c6d0fd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33819 } }
I20260924 00:19:17.607530 998 leader_election.cc:302] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 690196df77374d2682847acb2c6d0fd0; no voters:
I20260924 00:19:17.607715 998 leader_election.cc:288] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:19:17.607748 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:17.607883 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 930
I20260924 00:19:17.607998 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:19:17.608027 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:19:17.608086 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0 [term 1 LEADER]: Becoming Leader. State: Replica: 690196df77374d2682847acb2c6d0fd0, State: Running, Role: LEADER
I20260924 00:19:17.608206 392 external_mini_cluster.cc:1428] Setting key 2799faea867f4dc7345fcc41df2560f1
I20260924 00:19:17.608208 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690196df77374d2682847acb2c6d0fd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33819 } }
I20260924 00:19:17.608528 1001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 690196df77374d2682847acb2c6d0fd0. Latest consensus state: current_term: 1 leader_uuid: "690196df77374d2682847acb2c6d0fd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690196df77374d2682847acb2c6d0fd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33819 } } }
I20260924 00:19:17.608633 1001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:17.608539 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "690196df77374d2682847acb2c6d0fd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "690196df77374d2682847acb2c6d0fd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33819 } } }
I20260924 00:19:17.608837 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:17.609306 1009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:19:17.609843 1009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:19:17.611357 1009 catalog_manager.cc:1347] Generated new cluster ID: e846ba2b320d458c92823843cc2685e4
I20260924 00:19:17.611413 1009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:19:17 dist-test-slave-777x krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:17.620553 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:17.608939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57110 (local address 127.0.98.62:33819)
0924 00:19:17.609126 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:17.609137 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:17.609179 (+ 42us) server_negotiation.cc:407] Connection header received
0924 00:19:17.609321 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:17.609329 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:17.609536 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:17.610506 (+ 970us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:17.610604 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:17.611965 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:17.613046 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:17.613690 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:17.613817 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:17.616746 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:17.616751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:17.616751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:17.616765 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:17.618789 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:17.619235 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:17.619239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:17.619239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:17.619350 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:17.619560 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:17.619560 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:17.619561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:17.619870 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:17.620030 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:17.620144 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 00:19:17.620315 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":101,"threads_started":1}
I20260924 00:19:17.627766 1009 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:19:17.628526 1009 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:19:17.632066 1009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 690196df77374d2682847acb2c6d0fd0: Generated new TSK 0
I20260924 00:19:17.632293 1009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:19:17.667080 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33819
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:17.750751 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:17.750981 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:17.751008 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:17.752323 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:17.752372 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:17.752409 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:19:17.753890 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33819
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:17.754094 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:17.754284 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:17.759471 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:17.759470 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:17.759637 1024 server_base.cc:1034] running on GCE node
W20260924 00:19:17.759537 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:17.759967 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:17.760288 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:17.761475 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790209157761435 us; error 34 us; skew 500 ppm
Sep 24 00:19:17 dist-test-slave-777x krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:17.764451 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:19:17.764928 1024 webserver.cc:460] Webserver started at http://127.0.98.1:45805/ using document root <none> and password file <none>
I20260924 00:19:17.765130 1024 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:17.765175 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:17.765275 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:17.766390 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "d4c03df7a5dd4c689fbb44f0fcb27aa0"
format_stamp: "Formatted at 2026-09-24 00:19:17 on dist-test-slave-777x"
server_key: "bea8053e96356f53f70c39afd8e0387d"
server_key_iv: "a01d15c1708acdf43ec306a7ed3acebc"
server_key_version: "encryptionkey@0"
I20260924 00:19:17.766729 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d4c03df7a5dd4c689fbb44f0fcb27aa0"
format_stamp: "Formatted at 2026-09-24 00:19:17 on dist-test-slave-777x"
server_key: "bea8053e96356f53f70c39afd8e0387d"
server_key_iv: "a01d15c1708acdf43ec306a7ed3acebc"
server_key_version: "encryptionkey@0"
I20260924 00:19:17.768101 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 00:19:17.768959 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:17.769181 1024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:17.769260 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "d4c03df7a5dd4c689fbb44f0fcb27aa0"
format_stamp: "Formatted at 2026-09-24 00:19:17 on dist-test-slave-777x"
server_key: "bea8053e96356f53f70c39afd8e0387d"
server_key_iv: "a01d15c1708acdf43ec306a7ed3acebc"
server_key_version: "encryptionkey@0"
I20260924 00:19:17.769315 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:17.775760 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:17.778163 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:17.778276 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:17.778460 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:17.778782 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:17.778815 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:17.778846 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:17.778888 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:17.784834 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34991
I20260924 00:19:17.784921 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34991 every 8 connection(s)
I20260924 00:19:17.785248 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 00:19:17 dist-test-slave-777x krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:17.792092 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 1024
I20260924 00:19:17.792229 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:19:17.792413 392 external_mini_cluster.cc:1428] Setting key 94822f14bc1f4579dd261385f2ca1257
I20260924 00:19:17.795158 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:17.786013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38687 (local address 127.0.98.62:33819)
0924 00:19:17.786061 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:17.786063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:17.786257 (+ 194us) server_negotiation.cc:407] Connection header received
0924 00:19:17.786744 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:17.786746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:17.786763 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:17.786826 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:17.787800 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:17.788414 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:17.789523 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:17.789725 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:17.789791 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:17.791947 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:17.791948 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:17.791948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:17.791956 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:17.793591 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:17.794113 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:17.794114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:17.794114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:17.794166 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:17.794453 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:17.794454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:17.794455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:17.794603 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:17.794675 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:17.794940 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 00:19:17.795033 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 00:19:17.795192 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:17.785979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33819 (local address 127.0.98.1:38687)
0924 00:19:17.786134 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:17.786176 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:17.786650 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:17.786894 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:17.786898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:17.787476 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:17.787676 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:17.787681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:17.788519 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:17.788520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:17.789423 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:17.789426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:17.789807 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:17.789807 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:17.789944 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:17.790433 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:17.790440 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:17.791852 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:17.793672 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:17.793679 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:17.793681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:17.794038 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:17.794230 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:17.794231 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:17.794231 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:17.794377 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:17.794674 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:17.794679 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:17.794801 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 00:19:17.794900 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 00:19:17.795025 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260924 00:19:17.796062 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33819
I20260924 00:19:17.796196 1153 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:17.796427 1153 heartbeater.cc:511] Master 127.0.98.62:33819 requested a full tablet report, sending...
I20260924 00:19:17.797039 962 ts_manager.cc:194] Registered new tserver with Master: d4c03df7a5dd4c689fbb44f0fcb27aa0 (127.0.98.1:34991)
I20260924 00:19:17.797730 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38687
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:19:17.819634 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33819
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:17.901120 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:17.901341 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:17.901365 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:17.902474 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:17.902509 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:17.902535 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:19:17.903981 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33819
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:17.904218 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:17.904433 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:17.909880 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:17.909927 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:17.910089 1158 server_base.cc:1034] running on GCE node
W20260924 00:19:17.909894 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:17.910410 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:17.910712 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:17.911913 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790209157911904 us; error 35 us; skew 500 ppm
Sep 24 00:19:17 dist-test-slave-777x krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:17.914454 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:19:17.914947 1158 webserver.cc:460] Webserver started at http://127.0.98.2:42285/ using document root <none> and password file <none>
I20260924 00:19:17.915195 1158 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:17.915295 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:17.915455 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:17.916630 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "644b0b0ba2674c179f9af281ecf12638"
format_stamp: "Formatted at 2026-09-24 00:19:17 on dist-test-slave-777x"
server_key: "f69cd926a83065b2153888102d0214e4"
server_key_iv: "eaa8c38dd6907cb999f9d61bd92dc6d8"
server_key_version: "encryptionkey@0"
I20260924 00:19:17.916999 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "644b0b0ba2674c179f9af281ecf12638"
format_stamp: "Formatted at 2026-09-24 00:19:17 on dist-test-slave-777x"
server_key: "f69cd926a83065b2153888102d0214e4"
server_key_iv: "eaa8c38dd6907cb999f9d61bd92dc6d8"
server_key_version: "encryptionkey@0"
I20260924 00:19:17.918286 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:19:17.918975 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:17.919142 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:19:17.919245 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "644b0b0ba2674c179f9af281ecf12638"
format_stamp: "Formatted at 2026-09-24 00:19:17 on dist-test-slave-777x"
server_key: "f69cd926a83065b2153888102d0214e4"
server_key_iv: "eaa8c38dd6907cb999f9d61bd92dc6d8"
server_key_version: "encryptionkey@0"
I20260924 00:19:17.919317 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:17.955344 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:17.957665 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:17.957805 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:17.957998 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:17.958364 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:17.958405 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:17.958457 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:17.958487 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:17.964797 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42767
I20260924 00:19:17.964881 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42767 every 8 connection(s)
I20260924 00:19:17.965241 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 00:19:17 dist-test-slave-777x krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:17.974895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 1158
I20260924 00:19:17.974977 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:19:17.975148 392 external_mini_cluster.cc:1428] Setting key dcb6f30c821a4f983f12a23a07283ece
I20260924 00:19:17.975353 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:17.965991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36683 (local address 127.0.98.62:33819)
0924 00:19:17.966069 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:17.966072 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:17.966303 (+ 231us) server_negotiation.cc:407] Connection header received
0924 00:19:17.966857 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:17.966858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:17.966868 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:17.966922 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:17.967932 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:17.968519 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:17.969658 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:17.969866 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:17.969929 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:17.972255 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:17.972257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:17.972257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:17.972265 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:17.973754 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:17.974152 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:17.974153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:17.974153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:17.974200 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:17.974556 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:17.974557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:17.974557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:17.974661 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:17.974741 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:17.975107 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 00:19:17.975245 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 00:19:17.975602 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:17.965971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33819 (local address 127.0.98.2:36683)
0924 00:19:17.966145 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:17.966214 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:17.966741 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:17.966985 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:17.966988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:17.967615 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:17.967831 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:17.967836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:17.968613 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:17.968614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:17.969569 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:17.969571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:17.969987 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:17.969988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:17.970099 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:17.970565 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:17.970574 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:17.972097 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:17.973834 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:17.973841 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:17.973843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:17.974084 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:17.974290 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:17.974291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:17.974291 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:17.974431 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:17.974763 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:17.974769 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:17.974916 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 00:19:17.975096 (+ 180us) client_negotiation.cc:240] Negotiation successful
0924 00:19:17.975303 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":107,"threads_started":1}
I20260924 00:19:17.976135 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33819
I20260924 00:19:17.976257 1287 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:17.976446 1287 heartbeater.cc:511] Master 127.0.98.62:33819 requested a full tablet report, sending...
I20260924 00:19:17.976873 962 ts_manager.cc:194] Registered new tserver with Master: 644b0b0ba2674c179f9af281ecf12638 (127.0.98.2:42767)
I20260924 00:19:17.977458 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36683
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:19:18.002978 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33819
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:18.087383 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:18.087670 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:18.087711 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:18.089248 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:18.089344 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:18.089406 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:19:18.091037 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33819
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:18.091333 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:18.091588 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:18.097378 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:18.097378 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:18.097569 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:18.097880 1292 server_base.cc:1034] running on GCE node
I20260924 00:19:18.098114 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:18.098408 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:18.099592 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790209158099572 us; error 43 us; skew 500 ppm
Sep 24 00:19:18 dist-test-slave-777x krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:18.102320 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:19:18.102881 1292 webserver.cc:460] Webserver started at http://127.0.98.3:43259/ using document root <none> and password file <none>
I20260924 00:19:18.103111 1292 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:18.103178 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:18.103335 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:18.104605 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "49b31e2b38b541dab5f91f9eb1d3753a"
format_stamp: "Formatted at 2026-09-24 00:19:18 on dist-test-slave-777x"
server_key: "a7b2624881fc38f1fd61363e7b5c43bd"
server_key_iv: "cef87d0b7c5215586a2d5f8c83b2d6de"
server_key_version: "encryptionkey@0"
I20260924 00:19:18.104977 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "49b31e2b38b541dab5f91f9eb1d3753a"
format_stamp: "Formatted at 2026-09-24 00:19:18 on dist-test-slave-777x"
server_key: "a7b2624881fc38f1fd61363e7b5c43bd"
server_key_iv: "cef87d0b7c5215586a2d5f8c83b2d6de"
server_key_version: "encryptionkey@0"
I20260924 00:19:18.108031 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260924 00:19:18.109148 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:18.109386 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:18.109505 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "49b31e2b38b541dab5f91f9eb1d3753a"
format_stamp: "Formatted at 2026-09-24 00:19:18 on dist-test-slave-777x"
server_key: "a7b2624881fc38f1fd61363e7b5c43bd"
server_key_iv: "cef87d0b7c5215586a2d5f8c83b2d6de"
server_key_version: "encryptionkey@0"
I20260924 00:19:18.109591 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:18.123555 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:18.126076 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:18.126235 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:18.126497 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:18.126816 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:18.126852 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:18.126878 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:18.126890 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:18.133973 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43719
I20260924 00:19:18.134049 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43719 every 8 connection(s)
I20260924 00:19:18.134447 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:19:18.138165 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 1292
I20260924 00:19:18.138283 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:19:18.138473 392 external_mini_cluster.cc:1428] Setting key 8d984862abd612dbd74b1c1451766997
Sep 24 00:19:18 dist-test-slave-777x krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:18.145511 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:18.135382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55785 (local address 127.0.98.62:33819)
0924 00:19:18.135445 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:18.135447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:18.135685 (+ 238us) server_negotiation.cc:407] Connection header received
0924 00:19:18.136242 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:18.136243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:18.136257 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:18.136331 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:18.137384 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.137979 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.139625 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.139832 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.139891 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:18.142137 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:18.142138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:18.142139 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:18.142146 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:18.143795 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:18.144428 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:18.144429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:18.144429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:18.144493 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:18.144818 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:18.144819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:18.144819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:18.144953 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:18.145016 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:18.145362 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 00:19:18.145442 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 00:19:18.145668 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:18.135339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33819 (local address 127.0.98.3:55785)
0924 00:19:18.135511 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:18.135555 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:18.136070 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:18.136429 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:18.136433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:18.137046 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:18.137261 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:18.137266 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.138110 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.138111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:18.139549 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:18.139556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.139908 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.139908 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:18.140020 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:18.140610 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:18.140618 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:18.142043 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:18.143952 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:18.143959 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:18.143962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:18.144338 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:18.144581 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:18.144582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:18.144583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:18.144731 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:18.145043 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:18.145050 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:18.145203 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 00:19:18.145299 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 00:19:18.145445 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260924 00:19:18.146270 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33819
I20260924 00:19:18.146420 1421 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:18.146641 1421 heartbeater.cc:511] Master 127.0.98.62:33819 requested a full tablet report, sending...
I20260924 00:19:18.147037 962 ts_manager.cc:194] Registered new tserver with Master: 49b31e2b38b541dab5f91f9eb1d3753a (127.0.98.3:43719)
I20260924 00:19:18.147512 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55785
I20260924 00:19:18.150363 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:19:18.159619 1012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:18.152110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57116 (local address 127.0.98.62:33819)
0924 00:19:18.152243 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:18.152245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:18.152304 (+ 59us) server_negotiation.cc:407] Connection header received
0924 00:19:18.152359 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:18.152360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:18.152370 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:18.152411 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:18.153013 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.153571 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.154881 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.155072 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.155152 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:18.157035 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:18.157036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:18.157037 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:18.157044 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:18.158470 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:18.158858 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:18.158859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:18.158859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:18.158913 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:18.159093 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:18.159093 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:18.159093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:18.159242 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:18.159347 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:18.159449 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 00:19:18.159517 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 00:19:18.162102 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:19:18.163681 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:19:18.170709 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:18.166438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34320 (local address 127.0.98.3:43719)
0924 00:19:18.166796 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:18.166801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:18.166812 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:19:18.166903 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:18.166905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:18.166920 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:18.167018 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:18.167210 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.168071 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.169356 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.170180 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.170256 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:18.170315 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:18.170576 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 00:19:18.170623 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":79,"threads_started":1}
I20260924 00:19:18.170701 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:18.166384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34991 (local address 127.0.0.1:55560)
0924 00:19:18.166541 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:18.166581 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:18.166645 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:18.167342 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:18.167343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:18.167345 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:18.167388 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:18.167390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.168412 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.168413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:18.169569 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:18.169572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.170368 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.170369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:18.170451 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:18.170466 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 00:19:18.170562 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 00:19:18.170607 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":99,"threads_started":1}
I20260924 00:19:18.170701 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:18.166342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43719 (local address 127.0.0.1:34320)
0924 00:19:18.166538 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:18.166581 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:18.166645 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:18.167037 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:18.167041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:18.167043 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:18.167118 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:18.167120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.168228 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.168229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:18.169246 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:18.169248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.170352 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.170353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:18.170455 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:18.170468 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 00:19:18.170542 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 00:19:18.170595 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":75,"threads_started":1}
I20260924 00:19:18.170959 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:18.166638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55560 (local address 127.0.98.1:34991)
0924 00:19:18.167156 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:18.167161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:18.167172 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:19:18.167212 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:18.167213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:18.167229 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:18.167341 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:18.167453 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.168275 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.169662 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.170303 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.170370 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:18.170428 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:18.170843 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 00:19:18.170888 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":92,"threads_started":1}
I20260924 00:19:18.171773 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 047b53f189604743ae9b226aaf07c925 (DEFAULT_TABLE table=test-table [id=6a1a24d0723f40beb3600778a0d0a997]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:18.171916 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 047b53f189604743ae9b226aaf07c925 (DEFAULT_TABLE table=test-table [id=6a1a24d0723f40beb3600778a0d0a997]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:18.172223 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047b53f189604743ae9b226aaf07c925. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:18.172297 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047b53f189604743ae9b226aaf07c925. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:18.172884 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:18.166552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42767 (local address 127.0.0.1:59052)
0924 00:19:18.166960 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:18.166970 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:18.167026 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:18.168273 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:18.168274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:18.168276 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:18.168326 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:18.168328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.169620 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.169621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:18.171599 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:18.171602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.172614 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.172615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:18.172688 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:18.172693 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:19:18.172763 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 00:19:18.172806 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":71,"threads_started":1}
I20260924 00:19:18.172936 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:18.166763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59052 (local address 127.0.98.2:42767)
0924 00:19:18.166955 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:18.166960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:18.167014 (+ 54us) server_negotiation.cc:407] Connection header received
0924 00:19:18.167090 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:18.167092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:18.167106 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:18.167199 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:18.168414 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.169513 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.171692 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.172480 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.172556 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:18.172610 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:18.172792 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 00:19:18.172861 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":109,"threads_started":1}
I20260924 00:19:18.173962 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 047b53f189604743ae9b226aaf07c925 (DEFAULT_TABLE table=test-table [id=6a1a24d0723f40beb3600778a0d0a997]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:18.174345 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047b53f189604743ae9b226aaf07c925. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:18.175735 1438 tablet_bootstrap.cc:492] T 047b53f189604743ae9b226aaf07c925 P d4c03df7a5dd4c689fbb44f0fcb27aa0: Bootstrap starting.
I20260924 00:19:18.176332 1439 tablet_bootstrap.cc:492] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a: Bootstrap starting.
I20260924 00:19:18.176585 1438 tablet_bootstrap.cc:654] T 047b53f189604743ae9b226aaf07c925 P d4c03df7a5dd4c689fbb44f0fcb27aa0: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:18.177033 1439 tablet_bootstrap.cc:654] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:18.177065 1438 log.cc:824] T 047b53f189604743ae9b226aaf07c925 P d4c03df7a5dd4c689fbb44f0fcb27aa0: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:18.177673 1439 log.cc:824] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:18.177803 1438 tablet_bootstrap.cc:492] T 047b53f189604743ae9b226aaf07c925 P d4c03df7a5dd4c689fbb44f0fcb27aa0: No bootstrap required, opened a new log
I20260924 00:19:18.177877 1438 ts_tablet_manager.cc:1397] T 047b53f189604743ae9b226aaf07c925 P d4c03df7a5dd4c689fbb44f0fcb27aa0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:19:18.178048 1442 tablet_bootstrap.cc:492] T 047b53f189604743ae9b226aaf07c925 P 644b0b0ba2674c179f9af281ecf12638: Bootstrap starting.
I20260924 00:19:18.178722 1442 tablet_bootstrap.cc:654] T 047b53f189604743ae9b226aaf07c925 P 644b0b0ba2674c179f9af281ecf12638: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:18.179009 1439 tablet_bootstrap.cc:492] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a: No bootstrap required, opened a new log
I20260924 00:19:18.179085 1439 ts_tablet_manager.cc:1397] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:19:18.179332 1442 log.cc:824] T 047b53f189604743ae9b226aaf07c925 P 644b0b0ba2674c179f9af281ecf12638: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:18.180058 1442 tablet_bootstrap.cc:492] T 047b53f189604743ae9b226aaf07c925 P 644b0b0ba2674c179f9af281ecf12638: No bootstrap required, opened a new log
I20260924 00:19:18.180004 1438 raft_consensus.cc:348] T 047b53f189604743ae9b226aaf07c925 P d4c03df7a5dd4c689fbb44f0fcb27aa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49b31e2b38b541dab5f91f9eb1d3753a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43719 } } peers { permanent_uuid: "644b0b0ba2674c179f9af281ecf12638" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } } peers { permanent_uuid: "d4c03df7a5dd4c689fbb44f0fcb27aa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34991 } }
I20260924 00:19:18.180145 1442 ts_tablet_manager.cc:1397] T 047b53f189604743ae9b226aaf07c925 P 644b0b0ba2674c179f9af281ecf12638: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:19:18.180199 1438 raft_consensus.cc:374] T 047b53f189604743ae9b226aaf07c925 P d4c03df7a5dd4c689fbb44f0fcb27aa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:18.180244 1438 raft_consensus.cc:729] T 047b53f189604743ae9b226aaf07c925 P d4c03df7a5dd4c689fbb44f0fcb27aa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4c03df7a5dd4c689fbb44f0fcb27aa0, State: Initialized, Role: FOLLOWER
I20260924 00:19:18.180382 1438 consensus_queue.cc:260] T 047b53f189604743ae9b226aaf07c925 P d4c03df7a5dd4c689fbb44f0fcb27aa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49b31e2b38b541dab5f91f9eb1d3753a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43719 } } peers { permanent_uuid: "644b0b0ba2674c179f9af281ecf12638" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } } peers { permanent_uuid: "d4c03df7a5dd4c689fbb44f0fcb27aa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34991 } }
I20260924 00:19:18.181125 1438 ts_tablet_manager.cc:1428] T 047b53f189604743ae9b226aaf07c925 P d4c03df7a5dd4c689fbb44f0fcb27aa0: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260924 00:19:18.181254 1153 heartbeater.cc:503] Master 127.0.98.62:33819 was elected leader, sending a full tablet report...
I20260924 00:19:18.181514 1439 raft_consensus.cc:348] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49b31e2b38b541dab5f91f9eb1d3753a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43719 } } peers { permanent_uuid: "644b0b0ba2674c179f9af281ecf12638" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } } peers { permanent_uuid: "d4c03df7a5dd4c689fbb44f0fcb27aa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34991 } }
I20260924 00:19:18.181677 1439 raft_consensus.cc:374] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:18.181717 1439 raft_consensus.cc:729] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49b31e2b38b541dab5f91f9eb1d3753a, State: Initialized, Role: FOLLOWER
I20260924 00:19:18.181829 1439 consensus_queue.cc:260] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49b31e2b38b541dab5f91f9eb1d3753a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43719 } } peers { permanent_uuid: "644b0b0ba2674c179f9af281ecf12638" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } } peers { permanent_uuid: "d4c03df7a5dd4c689fbb44f0fcb27aa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34991 } }
I20260924 00:19:18.182027 1442 raft_consensus.cc:348] T 047b53f189604743ae9b226aaf07c925 P 644b0b0ba2674c179f9af281ecf12638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49b31e2b38b541dab5f91f9eb1d3753a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43719 } } peers { permanent_uuid: "644b0b0ba2674c179f9af281ecf12638" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } } peers { permanent_uuid: "d4c03df7a5dd4c689fbb44f0fcb27aa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34991 } }
I20260924 00:19:18.182231 1442 raft_consensus.cc:374] T 047b53f189604743ae9b226aaf07c925 P 644b0b0ba2674c179f9af281ecf12638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:18.182241 1439 ts_tablet_manager.cc:1428] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:19:18.182277 1442 raft_consensus.cc:729] T 047b53f189604743ae9b226aaf07c925 P 644b0b0ba2674c179f9af281ecf12638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 644b0b0ba2674c179f9af281ecf12638, State: Initialized, Role: FOLLOWER
I20260924 00:19:18.182476 1421 heartbeater.cc:503] Master 127.0.98.62:33819 was elected leader, sending a full tablet report...
I20260924 00:19:18.182430 1442 consensus_queue.cc:260] T 047b53f189604743ae9b226aaf07c925 P 644b0b0ba2674c179f9af281ecf12638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49b31e2b38b541dab5f91f9eb1d3753a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43719 } } peers { permanent_uuid: "644b0b0ba2674c179f9af281ecf12638" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } } peers { permanent_uuid: "d4c03df7a5dd4c689fbb44f0fcb27aa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34991 } }
I20260924 00:19:18.182749 1442 ts_tablet_manager.cc:1428] T 047b53f189604743ae9b226aaf07c925 P 644b0b0ba2674c179f9af281ecf12638: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:19:18.182799 1287 heartbeater.cc:503] Master 127.0.98.62:33819 was elected leader, sending a full tablet report...
W20260924 00:19:18.215842 1288 tablet.cc:2369] T 047b53f189604743ae9b226aaf07c925 P 644b0b0ba2674c179f9af281ecf12638: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:19:18.233985 1445 raft_consensus.cc:482] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:19:18.234118 1445 raft_consensus.cc:504] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49b31e2b38b541dab5f91f9eb1d3753a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43719 } } peers { permanent_uuid: "644b0b0ba2674c179f9af281ecf12638" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } } peers { permanent_uuid: "d4c03df7a5dd4c689fbb44f0fcb27aa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34991 } }
I20260924 00:19:18.234483 1445 leader_election.cc:288] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 644b0b0ba2674c179f9af281ecf12638 (127.0.98.2:42767), d4c03df7a5dd4c689fbb44f0fcb27aa0 (127.0.98.1:34991)
I20260924 00:19:18.237653 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:18.234755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34991 (local address 127.0.98.3:41379)
0924 00:19:18.234991 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:18.235000 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:18.235053 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:18.235282 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:18.235282 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:18.235284 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:18.235358 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:18.235360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.236019 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.236020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:18.236822 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:18.236825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.237391 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.237392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:18.237470 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:18.237473 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:19:18.237516 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 00:19:18.237546 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":155,"threads_started":1}
I20260924 00:19:18.237664 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:18.234873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41379 (local address 127.0.98.1:34991)
0924 00:19:18.234944 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:18.234946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:18.235064 (+ 118us) server_negotiation.cc:407] Connection header received
0924 00:19:18.235105 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:18.235106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:18.235119 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:18.235174 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:18.235431 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.235936 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.236938 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.237311 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.237366 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:18.237374 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:18.237545 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 00:19:18.237595 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 00:19:18.237720 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:18.234750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35967 (local address 127.0.98.2:42767)
0924 00:19:18.234800 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:18.234803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:18.234815 (+ 12us) server_negotiation.cc:407] Connection header received
0924 00:19:18.234853 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:18.234854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:18.234866 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:18.234928 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:18.235269 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.235809 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.236906 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.237328 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.237416 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:18.237426 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:18.237605 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 00:19:18.237644 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 00:19:18.237766 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:18.234675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42767 (local address 127.0.98.3:35967)
0924 00:19:18.234733 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:18.234746 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:18.234798 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:18.234964 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:18.234965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:18.234967 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:18.235054 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:18.235056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.235903 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.235904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:18.236808 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:18.236810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.237433 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.237433 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:18.237524 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:18.237528 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:19:18.237587 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 00:19:18.237623 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":2}
I20260924 00:19:18.238034 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047b53f189604743ae9b226aaf07c925" candidate_uuid: "49b31e2b38b541dab5f91f9eb1d3753a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4c03df7a5dd4c689fbb44f0fcb27aa0" is_pre_election: true
I20260924 00:19:18.238058 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047b53f189604743ae9b226aaf07c925" candidate_uuid: "49b31e2b38b541dab5f91f9eb1d3753a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644b0b0ba2674c179f9af281ecf12638" is_pre_election: true
I20260924 00:19:18.238191 1107 raft_consensus.cc:2454] T 047b53f189604743ae9b226aaf07c925 P d4c03df7a5dd4c689fbb44f0fcb27aa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49b31e2b38b541dab5f91f9eb1d3753a in term 0.
I20260924 00:19:18.238191 1241 raft_consensus.cc:2454] T 047b53f189604743ae9b226aaf07c925 P 644b0b0ba2674c179f9af281ecf12638 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49b31e2b38b541dab5f91f9eb1d3753a in term 0.
I20260924 00:19:18.238442 1310 leader_election.cc:302] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49b31e2b38b541dab5f91f9eb1d3753a, d4c03df7a5dd4c689fbb44f0fcb27aa0; no voters:
I20260924 00:19:18.238608 1445 raft_consensus.cc:2781] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:19:18.238665 1445 raft_consensus.cc:482] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:19:18.238682 1445 raft_consensus.cc:3037] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a [term 0 FOLLOWER]: Advancing to term 1
W20260924 00:19:18.286036 1154 tablet.cc:2369] T 047b53f189604743ae9b226aaf07c925 P d4c03df7a5dd4c689fbb44f0fcb27aa0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:19:18.298265 1445 raft_consensus.cc:504] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49b31e2b38b541dab5f91f9eb1d3753a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43719 } } peers { permanent_uuid: "644b0b0ba2674c179f9af281ecf12638" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } } peers { permanent_uuid: "d4c03df7a5dd4c689fbb44f0fcb27aa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34991 } }
I20260924 00:19:18.298514 1445 leader_election.cc:288] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a [CANDIDATE]: Term 1 election: Requested vote from peers 644b0b0ba2674c179f9af281ecf12638 (127.0.98.2:42767), d4c03df7a5dd4c689fbb44f0fcb27aa0 (127.0.98.1:34991)
I20260924 00:19:18.298779 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047b53f189604743ae9b226aaf07c925" candidate_uuid: "49b31e2b38b541dab5f91f9eb1d3753a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4c03df7a5dd4c689fbb44f0fcb27aa0"
I20260924 00:19:18.298842 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "047b53f189604743ae9b226aaf07c925" candidate_uuid: "49b31e2b38b541dab5f91f9eb1d3753a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644b0b0ba2674c179f9af281ecf12638"
I20260924 00:19:18.298908 1107 raft_consensus.cc:3037] T 047b53f189604743ae9b226aaf07c925 P d4c03df7a5dd4c689fbb44f0fcb27aa0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:18.298980 1241 raft_consensus.cc:3037] T 047b53f189604743ae9b226aaf07c925 P 644b0b0ba2674c179f9af281ecf12638 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:18.299841 1241 raft_consensus.cc:2454] T 047b53f189604743ae9b226aaf07c925 P 644b0b0ba2674c179f9af281ecf12638 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49b31e2b38b541dab5f91f9eb1d3753a in term 1.
I20260924 00:19:18.300127 1308 leader_election.cc:302] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49b31e2b38b541dab5f91f9eb1d3753a, 644b0b0ba2674c179f9af281ecf12638; no voters:
I20260924 00:19:18.300146 1107 raft_consensus.cc:2454] T 047b53f189604743ae9b226aaf07c925 P d4c03df7a5dd4c689fbb44f0fcb27aa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49b31e2b38b541dab5f91f9eb1d3753a in term 1.
I20260924 00:19:18.300343 1445 raft_consensus.cc:2781] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:18.300518 1445 raft_consensus.cc:686] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a [term 1 LEADER]: Becoming Leader. State: Replica: 49b31e2b38b541dab5f91f9eb1d3753a, State: Running, Role: LEADER
I20260924 00:19:18.300654 1445 consensus_queue.cc:237] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49b31e2b38b541dab5f91f9eb1d3753a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43719 } } peers { permanent_uuid: "644b0b0ba2674c179f9af281ecf12638" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } } peers { permanent_uuid: "d4c03df7a5dd4c689fbb44f0fcb27aa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34991 } }
I20260924 00:19:18.301704 961 catalog_manager.cc:5495] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a reported cstate change: term changed from 0 to 1, leader changed from <none> to 49b31e2b38b541dab5f91f9eb1d3753a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "49b31e2b38b541dab5f91f9eb1d3753a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49b31e2b38b541dab5f91f9eb1d3753a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "644b0b0ba2674c179f9af281ecf12638" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4c03df7a5dd4c689fbb44f0fcb27aa0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34991 } health_report { overall_health: UNKNOWN } } }
Sep 24 00:19:18 dist-test-slave-777x krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:19:18.325593 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:18.317738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55574 (local address 127.0.98.1:34991)
0924 00:19:18.317808 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:18.317810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:18.317904 (+ 94us) server_negotiation.cc:407] Connection header received
0924 00:19:18.317977 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:18.317977 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:18.317987 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:18.318038 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:18.318694 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.319174 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.319870 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.320054 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.320109 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:18.322525 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:18.322542 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:18.322542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:18.322554 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:18.324386 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:18.324746 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:18.324754 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:18.324755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:18.324813 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:18.325026 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:18.325027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:18.325027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:18.325175 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:18.325324 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:18.325401 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 00:19:18.325485 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 00:19:18 dist-test-slave-777x krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209158, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260924 00:19:18.385136 1422 tablet.cc:2369] T 047b53f189604743ae9b226aaf07c925 P 49b31e2b38b541dab5f91f9eb1d3753a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 00:19:18 dist-test-slave-777x krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209158, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:19:18.406559 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:18.398054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55584 (local address 127.0.98.1:34991)
0924 00:19:18.398112 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:18.398115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:18.398257 (+ 142us) server_negotiation.cc:407] Connection header received
0924 00:19:18.398323 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:18.398324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:18.398337 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:18.398425 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:18.399026 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.399925 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.400877 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.401287 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.401382 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:18.403886 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:18.403888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:18.403888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:18.403896 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:18.405240 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:18.405629 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:18.405630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:18.405631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:18.405708 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:18.405938 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:18.405939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:18.405939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:18.406043 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:18.406123 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:18.406284 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 00:19:18.406449 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 00:19:18 dist-test-slave-777x krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 00:19:18 dist-test-slave-777x krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:19:18.500155 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:18.492381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55592 (local address 127.0.98.1:34991)
0924 00:19:18.492448 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:18.492450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:18.492460 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:19:18.492492 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:18.492493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:18.492504 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:18.492579 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:18.493273 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.493768 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.494581 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:18.494770 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:18.494853 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:18.497452 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:18.497454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:18.497454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:18.497461 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:18.498761 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:18.499365 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:18.499367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:18.499367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:18.499431 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:18.499651 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:18.499652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:18.499652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:18.499822 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:18.499928 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:18.500014 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 00:19:18.500086 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 00:19:18.500715 1067 tablet_service.cc:2908] Rejecting scan request for tablet 047b53f189604743ae9b226aaf07c925: Uninitialized: clean time has not yet been initialized
I20260924 00:19:18.502154 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 1024
I20260924 00:19:18.507576 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 1158
I20260924 00:19:18.512025 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 1292
I20260924 00:19:18.516832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 930
2026-09-24T00:19:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2241 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:19:18 dist-test-slave-777x krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:19:18 dist-test-slave-777x krb5kdc[1462](info): set up 2 sockets
Sep 24 00:19:18 dist-test-slave-777x krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260924 00:19:19.603901 392 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:19:19 dist-test-slave-777x krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:19:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:19:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:19:19.727214 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38677
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:19.809338 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:19.809587 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:19.809613 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:19.810858 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:19:19.810909 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 00:19:19.810922 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:19.810935 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:19.810947 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:19:19.810961 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:19:19.812561 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38677
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:19.812788 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:19.812981 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:19:19.818331 1475 server_base.cc:1034] running on GCE node
W20260924 00:19:19.818305 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:19.818341 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:19.818305 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:19.819075 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:19.819432 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:19.820595 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790209159820575 us; error 30 us; skew 500 ppm
Sep 24 00:19:19 dist-test-slave-777x krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:19.830237 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:19:19.830735 1475 webserver.cc:460] Webserver started at http://127.0.98.62:34063/ using document root <none> and password file <none>
I20260924 00:19:19.830997 1475 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:19.831045 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:19.831210 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:19.832476 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "d7d1c925c95f4013a9d782c44266993f"
format_stamp: "Formatted at 2026-09-24 00:19:19 on dist-test-slave-777x"
server_key: "152c8bfd45d263d74ac5d2280b29dca4"
server_key_iv: "6194e842ad29128a5919cbadd844652b"
server_key_version: "encryptionkey@0"
I20260924 00:19:19.832844 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "d7d1c925c95f4013a9d782c44266993f"
format_stamp: "Formatted at 2026-09-24 00:19:19 on dist-test-slave-777x"
server_key: "152c8bfd45d263d74ac5d2280b29dca4"
server_key_iv: "6194e842ad29128a5919cbadd844652b"
server_key_version: "encryptionkey@0"
I20260924 00:19:19.834209 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:19:19.835081 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:19.835294 1475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:19.835448 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "d7d1c925c95f4013a9d782c44266993f"
format_stamp: "Formatted at 2026-09-24 00:19:19 on dist-test-slave-777x"
server_key: "152c8bfd45d263d74ac5d2280b29dca4"
server_key_iv: "6194e842ad29128a5919cbadd844652b"
server_key_version: "encryptionkey@0"
I20260924 00:19:19.835525 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:19.868057 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:19.870260 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:19.870453 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:19.875509 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38677
I20260924 00:19:19.875630 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38677 every 8 connection(s)
I20260924 00:19:19.875973 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:19:19.876691 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:19.878997 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f: Bootstrap starting.
I20260924 00:19:19.879555 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:19.879807 1553 log.cc:824] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:19.880537 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f: No bootstrap required, opened a new log
I20260924 00:19:19.882614 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d1c925c95f4013a9d782c44266993f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38677 } }
I20260924 00:19:19.882781 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:19.882826 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7d1c925c95f4013a9d782c44266993f, State: Initialized, Role: FOLLOWER
I20260924 00:19:19.882997 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d1c925c95f4013a9d782c44266993f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38677 } }
I20260924 00:19:19.883014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 1475
I20260924 00:19:19.883081 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:19:19.883121 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:19:19.883138 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:19:19.883172 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:19.883296 392 external_mini_cluster.cc:1428] Setting key 3f06a1d76ff849fd60eff8022103f68e
I20260924 00:19:19.883970 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d1c925c95f4013a9d782c44266993f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38677 } }
I20260924 00:19:19.884106 1553 leader_election.cc:302] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7d1c925c95f4013a9d782c44266993f; no voters:
I20260924 00:19:19.884307 1553 leader_election.cc:288] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:19:19.884336 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:19.884536 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f [term 1 LEADER]: Becoming Leader. State: Replica: d7d1c925c95f4013a9d782c44266993f, State: Running, Role: LEADER
I20260924 00:19:19.884559 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:19:19.884680 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d1c925c95f4013a9d782c44266993f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38677 } }
I20260924 00:19:19.885124 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7d1c925c95f4013a9d782c44266993f. Latest consensus state: current_term: 1 leader_uuid: "d7d1c925c95f4013a9d782c44266993f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d1c925c95f4013a9d782c44266993f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38677 } } }
I20260924 00:19:19.885201 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:19.885344 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7d1c925c95f4013a9d782c44266993f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7d1c925c95f4013a9d782c44266993f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38677 } } }
I20260924 00:19:19.885442 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:19.885838 1565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:19:19.886396 1565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:19:19.888026 1565 catalog_manager.cc:1347] Generated new cluster ID: f9a1a616177a4c1fa98281933fb43aa7
I20260924 00:19:19.888079 1565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:19:19 dist-test-slave-777x krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:19.893040 1565 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:19:19.893730 1565 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:19:19.895339 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:19.883878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35984 (local address 127.0.98.62:38677)
0924 00:19:19.884088 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:19.884098 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:19.884134 (+ 36us) server_negotiation.cc:407] Connection header received
0924 00:19:19.884290 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:19.884298 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:19.884358 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:19.885356 (+ 998us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:19.885482 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:19.886958 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:19.887876 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:19.888523 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:19.888620 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:19.891432 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:19.891436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:19.891437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:19.891450 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:19.893680 (+ 2230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:19.894102 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:19.894105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:19.894105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:19.894219 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:19.894451 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:19.894453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:19.894453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:19.894725 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:19.894861 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:19.894996 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 00:19:19.895140 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":95,"threads_started":1}
I20260924 00:19:19.900565 1565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f: Generated new TSK 0
I20260924 00:19:19.900836 1565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:19:19.935667 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38677
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:20.017433 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:20.017680 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:20.017719 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:20.018905 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:20.018985 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:20.019021 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:19:20.019060 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:19:20.020674 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:20.020962 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:20.021198 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:20.026523 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:20.026539 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:20.026544 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:20.026625 1579 server_base.cc:1034] running on GCE node
I20260924 00:19:20.026937 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:20.027206 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:20.028402 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790209160028400 us; error 43 us; skew 500 ppm
Sep 24 00:19:20 dist-test-slave-777x krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:20.031122 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:19:20.031639 1579 webserver.cc:460] Webserver started at http://127.0.98.1:45925/ using document root <none> and password file <none>
I20260924 00:19:20.031885 1579 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:20.031953 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:20.032102 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:20.033263 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "99391323ce9d4936bb69ea0e331929c3"
format_stamp: "Formatted at 2026-09-24 00:19:20 on dist-test-slave-777x"
server_key: "d7e15a28176edbfb63d127cbddf021ea"
server_key_iv: "8ab803f46c42b173bee75de19d48236a"
server_key_version: "encryptionkey@0"
I20260924 00:19:20.033631 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "99391323ce9d4936bb69ea0e331929c3"
format_stamp: "Formatted at 2026-09-24 00:19:20 on dist-test-slave-777x"
server_key: "d7e15a28176edbfb63d127cbddf021ea"
server_key_iv: "8ab803f46c42b173bee75de19d48236a"
server_key_version: "encryptionkey@0"
I20260924 00:19:20.035050 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:19:20.035997 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:20.036286 1579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:20.036399 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "99391323ce9d4936bb69ea0e331929c3"
format_stamp: "Formatted at 2026-09-24 00:19:20 on dist-test-slave-777x"
server_key: "d7e15a28176edbfb63d127cbddf021ea"
server_key_iv: "8ab803f46c42b173bee75de19d48236a"
server_key_version: "encryptionkey@0"
I20260924 00:19:20.036510 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:20.081912 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:20.084436 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:20.084661 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:20.085312 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:20.085397 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:20.085431 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:20.085449 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 00:19:20 dist-test-slave-777x krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:20.093533 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44767
I20260924 00:19:20.093605 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44767 every 8 connection(s)
I20260924 00:19:20.094028 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:19:20.096879 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.085782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45707 (local address 127.0.98.62:38677)
0924 00:19:20.085850 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.085853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.086197 (+ 344us) server_negotiation.cc:407] Connection header received
0924 00:19:20.087006 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.087008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.087025 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.087109 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:20.088101 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.088733 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.090434 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.090653 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.090748 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.093727 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:20.093729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:20.093729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:20.093737 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:20.095363 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:20.095766 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:20.095767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:20.095767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:20.095843 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:20.096119 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:20.096120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:20.096120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:20.096321 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:20.096377 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.096664 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.096811 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 00:19:20.096915 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.085798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38677 (local address 127.0.98.1:45707)
0924 00:19:20.086021 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:20.086076 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:20.086841 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:20.087203 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:20.087209 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:20.087648 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:20.087973 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.087981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.088808 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.088809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.090315 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.090319 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.090774 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.090775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.090946 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.091745 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:20.091755 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:20.093608 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:20.095451 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:20.095456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:20.095458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:20.095695 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:20.095955 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:20.095955 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:20.095956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:20.096060 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:20.096367 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:20.096370 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:20.096517 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 00:19:20.096606 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 00:19:20.096742 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":128,"threads_started":1}
I20260924 00:19:20.097551 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38677
I20260924 00:19:20.097668 1710 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:20.097857 1710 heartbeater.cc:511] Master 127.0.98.62:38677 requested a full tablet report, sending...
I20260924 00:19:20.098587 1506 ts_manager.cc:194] Registered new tserver with Master: 99391323ce9d4936bb69ea0e331929c3 (127.0.98.1:44767)
I20260924 00:19:20.099318 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45707
I20260924 00:19:20.101801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 1579
I20260924 00:19:20.101934 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:19:20.102116 392 external_mini_cluster.cc:1428] Setting key fdcb70023d44f1d149fb0de1f7da0bc0
I20260924 00:19:20.106402 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.098799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37651 (local address 127.0.98.62:38677)
0924 00:19:20.098838 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.098840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.099669 (+ 829us) server_negotiation.cc:407] Connection header received
0924 00:19:20.099730 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.099730 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.099739 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.099776 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:20.100542 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.101055 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.101762 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.102034 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.102104 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.103428 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:20.103430 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:20.103430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:20.103438 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:20.105185 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:20.105632 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:20.105632 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:20.105633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:20.105689 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:20.105948 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:20.105949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:20.105949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:20.106050 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:20.106101 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.106269 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.106334 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 00:19:20.106536 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.098669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38677 (local address 127.0.98.1:37651)
0924 00:19:20.099590 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:20.099602 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:20.099670 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:20.100215 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:20.100217 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:20.100408 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:20.100463 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.100465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.101151 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.101152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.101658 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.101659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.102106 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.102107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.102173 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.102900 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:20.102905 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:20.103329 (+ 424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:20.105282 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:20.105283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:20.105284 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:20.105551 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:20.105758 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:20.105766 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:20.105767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:20.105851 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:20.106115 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:20.106116 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:20.106168 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 00:19:20.106244 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 00:19:20.106371 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:19:20.129269 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38677
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:20.212581 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:20.212796 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:20.212821 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:20.213925 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:20.213959 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:20.213979 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:19:20.213997 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:19:20.215421 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:20.215612 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:20.215781 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:20.221066 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:20.221061 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:20.221061 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:20.221434 1719 server_base.cc:1034] running on GCE node
I20260924 00:19:20.221686 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:20.221951 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:20.223096 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790209160223086 us; error 33 us; skew 500 ppm
Sep 24 00:19:20 dist-test-slave-777x krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:20.225726 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:19:20.226221 1719 webserver.cc:460] Webserver started at http://127.0.98.2:45565/ using document root <none> and password file <none>
I20260924 00:19:20.226446 1719 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:20.226498 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:20.226650 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:20.227775 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "bbf99dc978e44d32a87bf1da6d371356"
format_stamp: "Formatted at 2026-09-24 00:19:20 on dist-test-slave-777x"
server_key: "2f8faf5a90380ecc101e1ac58f5ffebd"
server_key_iv: "5643c93ee999b3afba8dfab2d2cb42a9"
server_key_version: "encryptionkey@0"
I20260924 00:19:20.228149 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bbf99dc978e44d32a87bf1da6d371356"
format_stamp: "Formatted at 2026-09-24 00:19:20 on dist-test-slave-777x"
server_key: "2f8faf5a90380ecc101e1ac58f5ffebd"
server_key_iv: "5643c93ee999b3afba8dfab2d2cb42a9"
server_key_version: "encryptionkey@0"
I20260924 00:19:20.229666 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:20.230566 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:20.230769 1719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:20.230947 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "bbf99dc978e44d32a87bf1da6d371356"
format_stamp: "Formatted at 2026-09-24 00:19:20 on dist-test-slave-777x"
server_key: "2f8faf5a90380ecc101e1ac58f5ffebd"
server_key_iv: "5643c93ee999b3afba8dfab2d2cb42a9"
server_key_version: "encryptionkey@0"
I20260924 00:19:20.231029 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:20.247957 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:20.250452 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:20.250617 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:20.251214 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:20.251251 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:20.251276 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:20.251288 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:20.258351 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45617
I20260924 00:19:20.258433 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45617 every 8 connection(s)
I20260924 00:19:20.258754 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 00:19:20 dist-test-slave-777x krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:20.263118 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.251903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50375 (local address 127.0.98.62:38677)
0924 00:19:20.251955 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.251957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.252191 (+ 234us) server_negotiation.cc:407] Connection header received
0924 00:19:20.252806 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.252808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.252822 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.252923 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:20.253918 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.254894 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.256518 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.256740 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.256833 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.259604 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:20.259606 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:20.259606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:20.259616 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:20.261611 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:20.262063 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:20.262065 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:20.262065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:20.262127 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:20.262408 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:20.262410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:20.262410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:20.262546 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:20.262625 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.262930 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.263022 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:19:20.263329 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.251898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38677 (local address 127.0.98.2:50375)
0924 00:19:20.252052 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:20.252089 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:20.252690 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:20.253011 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:20.253017 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:20.253420 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:20.253767 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.253772 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.255002 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.255003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.256390 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.256393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.256842 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.256843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.256992 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.257687 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:20.257699 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:20.259495 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:20.261707 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:20.261717 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:20.261719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:20.261975 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:20.262214 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:20.262215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:20.262215 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:20.262337 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:20.262615 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:20.262621 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:20.262790 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 00:19:20.262910 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 00:19:20.263080 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260924 00:19:20.263852 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38677
I20260924 00:19:20.263942 1850 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:20.264134 1850 heartbeater.cc:511] Master 127.0.98.62:38677 requested a full tablet report, sending...
I20260924 00:19:20.264122 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 1719
I20260924 00:19:20.264276 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:19:20.264477 392 external_mini_cluster.cc:1428] Setting key 05a58570ba1224e63a3430efa575d497
I20260924 00:19:20.264953 1506 ts_manager.cc:194] Registered new tserver with Master: bbf99dc978e44d32a87bf1da6d371356 (127.0.98.2:45617)
I20260924 00:19:20.265468 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50375
I20260924 00:19:20.271868 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.264944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38677 (local address 127.0.98.2:51789)
0924 00:19:20.265070 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:20.265081 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:20.265138 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:20.265256 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:20.265257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:20.265448 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:20.265511 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.265513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.266490 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.266491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.267440 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.267443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.267953 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.267954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.268007 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.268577 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:20.268582 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:20.268923 (+ 341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:20.270625 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:20.270627 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:20.270627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:20.270821 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:20.271066 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:20.271067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:20.271068 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:20.271132 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:20.271484 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:20.271486 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:20.271539 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 00:19:20.271638 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 00:19:20.271755 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260924 00:19:20.271863 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.264982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51789 (local address 127.0.98.62:38677)
0924 00:19:20.265015 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.265017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.265128 (+ 111us) server_negotiation.cc:407] Connection header received
0924 00:19:20.265185 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.265186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.265197 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.265251 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:20.265601 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.266367 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.267569 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.267873 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.267947 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.269049 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:20.269051 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:20.269051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:20.269060 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:20.270506 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:20.270921 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:20.270922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:20.270923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:20.270992 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:20.271222 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:20.271223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:20.271223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:20.271390 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:20.271481 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.271666 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.271753 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:19:20.293465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38677
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:20.385545 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:20.385783 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:20.385812 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:20.387256 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:20.387315 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:20.387339 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:19:20.387398 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:19:20.389178 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:20.389467 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:20.389734 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:20.395217 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:20.395332 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:20.395550 1859 server_base.cc:1034] running on GCE node
W20260924 00:19:20.395373 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:20.395797 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:20.396075 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:20.397311 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790209160397323 us; error 58 us; skew 500 ppm
Sep 24 00:19:20 dist-test-slave-777x krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:20.400280 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:19:20.400866 1859 webserver.cc:460] Webserver started at http://127.0.98.3:38981/ using document root <none> and password file <none>
I20260924 00:19:20.401185 1859 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:20.401253 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:20.401410 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:20.402539 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "9fe527fb953047a8970207a900a36a3e"
format_stamp: "Formatted at 2026-09-24 00:19:20 on dist-test-slave-777x"
server_key: "aca20363a43a48eae76a15a394720c35"
server_key_iv: "8c7ddbeead90ee0b0bb3fc4d6fc1377e"
server_key_version: "encryptionkey@0"
I20260924 00:19:20.402951 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9fe527fb953047a8970207a900a36a3e"
format_stamp: "Formatted at 2026-09-24 00:19:20 on dist-test-slave-777x"
server_key: "aca20363a43a48eae76a15a394720c35"
server_key_iv: "8c7ddbeead90ee0b0bb3fc4d6fc1377e"
server_key_version: "encryptionkey@0"
I20260924 00:19:20.404477 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:19:20.405244 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:20.405443 1859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:19:20.405539 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "9fe527fb953047a8970207a900a36a3e"
format_stamp: "Formatted at 2026-09-24 00:19:20 on dist-test-slave-777x"
server_key: "aca20363a43a48eae76a15a394720c35"
server_key_iv: "8c7ddbeead90ee0b0bb3fc4d6fc1377e"
server_key_version: "encryptionkey@0"
I20260924 00:19:20.405597 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:20.414803 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:20.417284 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:20.417447 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:20.417984 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:20.418025 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:20.418049 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:20.418061 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:20.425254 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38631
I20260924 00:19:20.425477 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38631 every 8 connection(s)
Sep 24 00:19:20 dist-test-slave-777x krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209160, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:20.425688 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:19:20.428647 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 1859
I20260924 00:19:20.428768 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:19:20.428959 392 external_mini_cluster.cc:1428] Setting key 868829498e1062c0cd403f89be58261f
I20260924 00:19:20.429473 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.418716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37463 (local address 127.0.98.62:38677)
0924 00:19:20.418786 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.418788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.419016 (+ 228us) server_negotiation.cc:407] Connection header received
0924 00:19:20.419727 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.419729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.419745 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.419829 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:20.420926 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.421787 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.422987 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.423360 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.423471 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.426172 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:20.426174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:20.426174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:20.426182 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:20.427853 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:20.428331 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:20.428333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:20.428333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:20.428400 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:20.428687 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:20.428689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:20.428689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:20.428864 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:20.428984 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.429288 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.429378 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 00:19:20.429708 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.418690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38677 (local address 127.0.98.3:37463)
0924 00:19:20.418877 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:20.418921 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:20.419594 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:20.419920 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:20.419926 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:20.420412 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:20.420792 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.420799 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.421914 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.421915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.422903 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.422905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.423481 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.423482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.423627 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.424284 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:20.424293 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:20.426018 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:20.427944 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:20.427949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:20.427951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:20.428247 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:20.428479 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:20.428480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:20.428480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:20.428596 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:20.428984 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:20.428991 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:20.429154 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 00:19:20.429253 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 00:19:20.429456 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":113,"threads_started":1}
I20260924 00:19:20.430224 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38677
I20260924 00:19:20.430320 1990 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:20.430513 1990 heartbeater.cc:511] Master 127.0.98.62:38677 requested a full tablet report, sending...
I20260924 00:19:20.430934 1506 ts_manager.cc:194] Registered new tserver with Master: 9fe527fb953047a8970207a900a36a3e (127.0.98.3:38631)
I20260924 00:19:20.431321 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37463
I20260924 00:19:20.436898 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.431462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41365 (local address 127.0.98.62:38677)
0924 00:19:20.431505 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.431507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.431602 (+ 95us) server_negotiation.cc:407] Connection header received
0924 00:19:20.431658 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.431659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.431669 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.431729 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:20.432033 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.432640 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.433377 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.433567 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.433636 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.434526 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:20.434527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:20.434528 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:20.434533 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:20.436017 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:20.436356 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:20.436357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:20.436357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:20.436402 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:20.436554 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:20.436554 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:20.436554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:20.436639 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:20.436683 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.436786 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.436837 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:19:20.436998 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.431408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38677 (local address 127.0.98.3:41365)
0924 00:19:20.431539 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:20.431550 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:20.431606 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:20.431738 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:20.431738 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:20.431904 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:20.431956 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.431957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.432726 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.432727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.433285 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.433287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.433646 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.433646 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.433692 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.434172 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:20.434176 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:20.434451 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:20.436088 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:20.436088 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:20.436089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:20.436298 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:20.436455 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:20.436455 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:20.436455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:20.436516 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:20.436687 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:20.436687 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:20.436720 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 00:19:20.436768 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 00:19:20.436842 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260924 00:19:20.441099 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 00:19:20 dist-test-slave-777x krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209160, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 00:19:20 dist-test-slave-777x krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209160, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:20.476392 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.467419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35992 (local address 127.0.98.62:38677)
0924 00:19:20.467485 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.467487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.467498 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:19:20.467537 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.467538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.467566 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.467656 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:20.468595 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.469516 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.470362 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.470587 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.470660 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.473204 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:20.473206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:20.473206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:20.473215 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:20.474909 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:20.475377 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:20.475378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:20.475379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:20.475462 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:20.475707 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:20.475709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:20.475709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:20.475873 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:20.475976 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.476144 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.476257 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:19:20.478415 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:19:20.480109 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:19:20.487143 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.482826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45617 (local address 127.0.0.1:43432)
0924 00:19:20.483013 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:20.483027 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:20.483088 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:20.483503 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:20.483504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:20.483506 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:20.483563 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.483565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.484587 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.484588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.485868 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.485870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.486802 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.486803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.486892 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.486909 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 00:19:20.486984 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 00:19:20.487047 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":137,"threads_started":1}
I20260924 00:19:20.487236 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.482722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51552 (local address 127.0.98.3:38631)
0924 00:19:20.483170 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.483175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.483186 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:19:20.483234 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.483237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.483255 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.483577 (+ 322us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:20.483645 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.484934 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.486169 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.486871 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.486948 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.487019 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.487092 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.487133 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":174,"threads_started":1}
I20260924 00:19:20.487475 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.482971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43432 (local address 127.0.98.2:45617)
0924 00:19:20.483322 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.483328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.483339 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:19:20.483380 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.483382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.483399 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.483637 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:20.483684 (+ 47us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.484487 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.485964 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.486695 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.486788 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.486877 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.487283 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.487344 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":79,"threads_started":1}
I20260924 00:19:20.487587 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.483101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49222 (local address 127.0.98.1:44767)
0924 00:19:20.483263 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.483267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.483277 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:19:20.483317 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.483319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.483335 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.483415 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:20.483614 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.484987 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.486404 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.487119 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.487183 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.487254 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.487444 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.487481 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260924 00:19:20.488045 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.482591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38631 (local address 127.0.0.1:51552)
0924 00:19:20.482745 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:20.482786 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:20.482849 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:20.483401 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:20.483405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:20.483407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:20.483490 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.483492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.485028 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.485029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.486082 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.486083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.486949 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.486950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.487012 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.487015 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:19:20.487077 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 00:19:20.487111 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260924 00:19:20.488243 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.482602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44767 (local address 127.0.0.1:49222)
0924 00:19:20.482995 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:20.483006 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:20.483048 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:20.483483 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:20.483483 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:20.483485 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:20.483521 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.483523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.485077 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.485078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.486310 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.486313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.487289 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.487290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.487360 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.487363 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:19:20.487416 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 00:19:20.487453 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":86,"threads_started":1}
I20260924 00:19:20.488521 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 95437b9d6274489eb52090591831ed6b (DEFAULT_TABLE table=test-table [id=cbc8869470534aefad07ddbd40bb71b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:20.488732 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 95437b9d6274489eb52090591831ed6b (DEFAULT_TABLE table=test-table [id=cbc8869470534aefad07ddbd40bb71b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:20.488878 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95437b9d6274489eb52090591831ed6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:20.489110 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95437b9d6274489eb52090591831ed6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:20.491967 2013 tablet_bootstrap.cc:492] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356: Bootstrap starting.
I20260924 00:19:20.492326 2014 tablet_bootstrap.cc:492] T 95437b9d6274489eb52090591831ed6b P 9fe527fb953047a8970207a900a36a3e: Bootstrap starting.
I20260924 00:19:20.492681 2013 tablet_bootstrap.cc:654] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:20.492745 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 95437b9d6274489eb52090591831ed6b (DEFAULT_TABLE table=test-table [id=cbc8869470534aefad07ddbd40bb71b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:20.493057 2014 tablet_bootstrap.cc:654] T 95437b9d6274489eb52090591831ed6b P 9fe527fb953047a8970207a900a36a3e: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:20.493057 2013 log.cc:824] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:20.493127 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95437b9d6274489eb52090591831ed6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:20.493460 2014 log.cc:824] T 95437b9d6274489eb52090591831ed6b P 9fe527fb953047a8970207a900a36a3e: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:20.493913 2013 tablet_bootstrap.cc:492] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356: No bootstrap required, opened a new log
I20260924 00:19:20.493973 2013 ts_tablet_manager.cc:1397] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:19:20.494199 2014 tablet_bootstrap.cc:492] T 95437b9d6274489eb52090591831ed6b P 9fe527fb953047a8970207a900a36a3e: No bootstrap required, opened a new log
I20260924 00:19:20.494266 2014 ts_tablet_manager.cc:1397] T 95437b9d6274489eb52090591831ed6b P 9fe527fb953047a8970207a900a36a3e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:19:20.496115 2013 raft_consensus.cc:348] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } }
I20260924 00:19:20.496315 2013 raft_consensus.cc:374] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:20.496353 2013 raft_consensus.cc:729] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf99dc978e44d32a87bf1da6d371356, State: Initialized, Role: FOLLOWER
I20260924 00:19:20.496503 2013 consensus_queue.cc:260] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } }
I20260924 00:19:20.496423 2014 raft_consensus.cc:348] T 95437b9d6274489eb52090591831ed6b P 9fe527fb953047a8970207a900a36a3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } }
I20260924 00:19:20.496620 2014 raft_consensus.cc:374] T 95437b9d6274489eb52090591831ed6b P 9fe527fb953047a8970207a900a36a3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:20.496649 2014 raft_consensus.cc:729] T 95437b9d6274489eb52090591831ed6b P 9fe527fb953047a8970207a900a36a3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe527fb953047a8970207a900a36a3e, State: Initialized, Role: FOLLOWER
I20260924 00:19:20.496795 2013 ts_tablet_manager.cc:1428] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:19:20.496845 1850 heartbeater.cc:503] Master 127.0.98.62:38677 was elected leader, sending a full tablet report...
I20260924 00:19:20.496805 2014 consensus_queue.cc:260] T 95437b9d6274489eb52090591831ed6b P 9fe527fb953047a8970207a900a36a3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } }
I20260924 00:19:20.497041 2014 ts_tablet_manager.cc:1428] T 95437b9d6274489eb52090591831ed6b P 9fe527fb953047a8970207a900a36a3e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:19:20.497217 1990 heartbeater.cc:503] Master 127.0.98.62:38677 was elected leader, sending a full tablet report...
I20260924 00:19:20.497702 2019 tablet_bootstrap.cc:492] T 95437b9d6274489eb52090591831ed6b P 99391323ce9d4936bb69ea0e331929c3: Bootstrap starting.
I20260924 00:19:20.498428 2019 tablet_bootstrap.cc:654] T 95437b9d6274489eb52090591831ed6b P 99391323ce9d4936bb69ea0e331929c3: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:20.498814 2019 log.cc:824] T 95437b9d6274489eb52090591831ed6b P 99391323ce9d4936bb69ea0e331929c3: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:20.499605 2019 tablet_bootstrap.cc:492] T 95437b9d6274489eb52090591831ed6b P 99391323ce9d4936bb69ea0e331929c3: No bootstrap required, opened a new log
I20260924 00:19:20.499665 2019 ts_tablet_manager.cc:1397] T 95437b9d6274489eb52090591831ed6b P 99391323ce9d4936bb69ea0e331929c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:19:20.501236 2019 raft_consensus.cc:348] T 95437b9d6274489eb52090591831ed6b P 99391323ce9d4936bb69ea0e331929c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } }
I20260924 00:19:20.501377 2019 raft_consensus.cc:374] T 95437b9d6274489eb52090591831ed6b P 99391323ce9d4936bb69ea0e331929c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:20.501402 2019 raft_consensus.cc:729] T 95437b9d6274489eb52090591831ed6b P 99391323ce9d4936bb69ea0e331929c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99391323ce9d4936bb69ea0e331929c3, State: Initialized, Role: FOLLOWER
I20260924 00:19:20.501484 2019 consensus_queue.cc:260] T 95437b9d6274489eb52090591831ed6b P 99391323ce9d4936bb69ea0e331929c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } }
I20260924 00:19:20.501680 2019 ts_tablet_manager.cc:1428] T 95437b9d6274489eb52090591831ed6b P 99391323ce9d4936bb69ea0e331929c3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:19:20.501729 1710 heartbeater.cc:503] Master 127.0.98.62:38677 was elected leader, sending a full tablet report...
W20260924 00:19:20.509409 1851 tablet.cc:2369] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:19:20.594956 1711 tablet.cc:2369] T 95437b9d6274489eb52090591831ed6b P 99391323ce9d4936bb69ea0e331929c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:19:20.668819 2017 raft_consensus.cc:482] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:19:20.668951 2017 raft_consensus.cc:504] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } }
I20260924 00:19:20.669384 2017 leader_election.cc:288] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fe527fb953047a8970207a900a36a3e (127.0.98.3:38631), 99391323ce9d4936bb69ea0e331929c3 (127.0.98.1:44767)
I20260924 00:19:20.672823 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.669650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57045 (local address 127.0.98.1:44767)
0924 00:19:20.669694 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.669697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.669708 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:19:20.669783 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.669785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.669799 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.669857 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:20.670068 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.670593 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.671863 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.672406 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.672487 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.672499 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.672693 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.672737 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:19:20.672838 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.669579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44767 (local address 127.0.98.2:57045)
0924 00:19:20.669635 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:20.669648 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:20.669699 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:20.669910 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:20.669911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:20.669913 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:20.669985 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.669986 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.670694 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.670695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.671763 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.671766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.672500 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.672501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.672605 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.672610 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:19:20.672672 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 00:19:20.672725 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 00:19:20.673285 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95437b9d6274489eb52090591831ed6b" candidate_uuid: "bbf99dc978e44d32a87bf1da6d371356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99391323ce9d4936bb69ea0e331929c3" is_pre_election: true
I20260924 00:19:20.673446 1664 raft_consensus.cc:2454] T 95437b9d6274489eb52090591831ed6b P 99391323ce9d4936bb69ea0e331929c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf99dc978e44d32a87bf1da6d371356 in term 0.
I20260924 00:19:20.673692 1738 leader_election.cc:302] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99391323ce9d4936bb69ea0e331929c3, bbf99dc978e44d32a87bf1da6d371356; no voters:
I20260924 00:19:20.673874 2017 raft_consensus.cc:2781] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:19:20.673954 2017 raft_consensus.cc:482] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:19:20.673990 2017 raft_consensus.cc:3037] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:20.674679 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.669603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38631 (local address 127.0.98.2:35439)
0924 00:19:20.669796 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:20.669807 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:20.669854 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:20.670718 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:20.670719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:20.670721 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:20.671132 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.671135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.672203 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.672204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.673389 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.673392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.674364 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.674365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.674440 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.674444 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:19:20.674516 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 00:19:20.674562 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":122,"threads_started":1}
I20260924 00:19:20.674676 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.669868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35439 (local address 127.0.98.3:38631)
0924 00:19:20.669919 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.669921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.669932 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:19:20.669962 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.669963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.669975 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.670068 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:20.671250 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.672077 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.673555 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.674242 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.674328 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.674340 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.674534 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.674583 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 00:19:20.674993 2017 raft_consensus.cc:504] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } }
I20260924 00:19:20.675194 2017 leader_election.cc:288] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356 [CANDIDATE]: Term 1 election: Requested vote from peers 9fe527fb953047a8970207a900a36a3e (127.0.98.3:38631), 99391323ce9d4936bb69ea0e331929c3 (127.0.98.1:44767)
I20260924 00:19:20.675240 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95437b9d6274489eb52090591831ed6b" candidate_uuid: "bbf99dc978e44d32a87bf1da6d371356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe527fb953047a8970207a900a36a3e" is_pre_election: true
I20260924 00:19:20.675420 1944 raft_consensus.cc:2454] T 95437b9d6274489eb52090591831ed6b P 9fe527fb953047a8970207a900a36a3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbf99dc978e44d32a87bf1da6d371356 in term 0.
I20260924 00:19:20.675457 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95437b9d6274489eb52090591831ed6b" candidate_uuid: "bbf99dc978e44d32a87bf1da6d371356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99391323ce9d4936bb69ea0e331929c3"
I20260924 00:19:20.675522 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95437b9d6274489eb52090591831ed6b" candidate_uuid: "bbf99dc978e44d32a87bf1da6d371356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe527fb953047a8970207a900a36a3e"
I20260924 00:19:20.675545 1664 raft_consensus.cc:3037] T 95437b9d6274489eb52090591831ed6b P 99391323ce9d4936bb69ea0e331929c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:20.675603 1943 raft_consensus.cc:3037] T 95437b9d6274489eb52090591831ed6b P 9fe527fb953047a8970207a900a36a3e [term 0 FOLLOWER]: Advancing to term 1
W20260924 00:19:20.676579 1991 tablet.cc:2369] T 95437b9d6274489eb52090591831ed6b P 9fe527fb953047a8970207a900a36a3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:19:20.676940 1664 raft_consensus.cc:2454] T 95437b9d6274489eb52090591831ed6b P 99391323ce9d4936bb69ea0e331929c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf99dc978e44d32a87bf1da6d371356 in term 1.
I20260924 00:19:20.677053 1943 raft_consensus.cc:2454] T 95437b9d6274489eb52090591831ed6b P 9fe527fb953047a8970207a900a36a3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbf99dc978e44d32a87bf1da6d371356 in term 1.
I20260924 00:19:20.677273 1735 leader_election.cc:302] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fe527fb953047a8970207a900a36a3e, bbf99dc978e44d32a87bf1da6d371356; no voters:
I20260924 00:19:20.677416 2017 raft_consensus.cc:2781] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:20.677649 2017 raft_consensus.cc:686] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356 [term 1 LEADER]: Becoming Leader. State: Replica: bbf99dc978e44d32a87bf1da6d371356, State: Running, Role: LEADER
I20260924 00:19:20.677760 2017 consensus_queue.cc:237] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } }
I20260924 00:19:20.678722 1507 catalog_manager.cc:5495] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbf99dc978e44d32a87bf1da6d371356 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bbf99dc978e44d32a87bf1da6d371356" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } health_report { overall_health: HEALTHY } } }
I20260924 00:19:20.716034 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.712762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43446 (local address 127.0.98.2:45617)
0924 00:19:20.712869 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.712872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.712886 (+ 14us) server_negotiation.cc:407] Connection header received
0924 00:19:20.712989 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.712991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.713018 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.713099 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:20.713248 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.714121 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.714935 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.715308 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.715407 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.715605 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:20.715755 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:20.715849 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.715894 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.715921 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 00:19:20.718171 1664 raft_consensus.cc:1261] T 95437b9d6274489eb52090591831ed6b P 99391323ce9d4936bb69ea0e331929c3 [term 1 FOLLOWER]: Refusing update from remote peer bbf99dc978e44d32a87bf1da6d371356: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:20.718197 1943 raft_consensus.cc:1261] T 95437b9d6274489eb52090591831ed6b P 9fe527fb953047a8970207a900a36a3e [term 1 FOLLOWER]: Refusing update from remote peer bbf99dc978e44d32a87bf1da6d371356: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:20.718623 2017 consensus_queue.cc:1035] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:20.718788 2023 consensus_queue.cc:1035] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:20.721434 2026 mvcc.cc:205] Tried to move back new op lower bound from 7332696722298126336 to 7332696722136805376. Current Snapshot: MvccSnapshot[applied={T|T < 7332696722298126336}]
I20260924 00:19:20.722102 2028 mvcc.cc:205] Tried to move back new op lower bound from 7332696722298126336 to 7332696722136805376. Current Snapshot: MvccSnapshot[applied={T|T < 7332696722298126336}]
I20260924 00:19:20.725858 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:20.725981 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:20.726430 2027 mvcc.cc:205] Tried to move back new op lower bound from 7332696722298126336 to 7332696722136805376. Current Snapshot: MvccSnapshot[applied={T|T < 7332696722298126336}]
I20260924 00:19:20.726909 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f: Sending DeleteTablet for 3 replicas of tablet 95437b9d6274489eb52090591831ed6b
I20260924 00:19:20.727587 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 95437b9d6274489eb52090591831ed6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51552
I20260924 00:19:20.727851 2035 tablet_replica.cc:330] T 95437b9d6274489eb52090591831ed6b P 9fe527fb953047a8970207a900a36a3e: stopping tablet replica
I20260924 00:19:20.727861 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 95437b9d6274489eb52090591831ed6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49222
I20260924 00:19:20.727969 2035 raft_consensus.cc:2229] T 95437b9d6274489eb52090591831ed6b P 9fe527fb953047a8970207a900a36a3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:20.728050 2035 raft_consensus.cc:2258] T 95437b9d6274489eb52090591831ed6b P 9fe527fb953047a8970207a900a36a3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:20.728375 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 95437b9d6274489eb52090591831ed6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43432
I20260924 00:19:20.728696 2036 tablet_replica.cc:330] T 95437b9d6274489eb52090591831ed6b P 99391323ce9d4936bb69ea0e331929c3: stopping tablet replica
I20260924 00:19:20.728858 2036 raft_consensus.cc:2229] T 95437b9d6274489eb52090591831ed6b P 99391323ce9d4936bb69ea0e331929c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:20.728861 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:19:20.729022 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:19:20.729123 2036 raft_consensus.cc:2258] T 95437b9d6274489eb52090591831ed6b P 99391323ce9d4936bb69ea0e331929c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:20.730751 1924 tablet_service.cc:1467] Processing CreateTablet for tablet ab2c4866567c4ffb9f0cdc2eeca20532 (DEFAULT_TABLE table=test-table [id=0c23bdef146b4d99a2c550305e448002]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:20.730923 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab2c4866567c4ffb9f0cdc2eeca20532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:20.732498 2014 tablet_bootstrap.cc:492] T ab2c4866567c4ffb9f0cdc2eeca20532 P 9fe527fb953047a8970207a900a36a3e: Bootstrap starting.
I20260924 00:19:20.733000 2014 tablet_bootstrap.cc:654] T ab2c4866567c4ffb9f0cdc2eeca20532 P 9fe527fb953047a8970207a900a36a3e: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:20.733772 1644 tablet_service.cc:1467] Processing CreateTablet for tablet ab2c4866567c4ffb9f0cdc2eeca20532 (DEFAULT_TABLE table=test-table [id=0c23bdef146b4d99a2c550305e448002]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:20.733876 1784 tablet_service.cc:1467] Processing CreateTablet for tablet ab2c4866567c4ffb9f0cdc2eeca20532 (DEFAULT_TABLE table=test-table [id=0c23bdef146b4d99a2c550305e448002]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:20.733907 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab2c4866567c4ffb9f0cdc2eeca20532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:20.734000 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab2c4866567c4ffb9f0cdc2eeca20532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:20.735565 2019 tablet_bootstrap.cc:492] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3: Bootstrap starting.
I20260924 00:19:20.736009 2019 tablet_bootstrap.cc:654] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:20.736411 2013 tablet_bootstrap.cc:492] T ab2c4866567c4ffb9f0cdc2eeca20532 P bbf99dc978e44d32a87bf1da6d371356: Bootstrap starting.
I20260924 00:19:20.737303 2013 tablet_bootstrap.cc:654] T ab2c4866567c4ffb9f0cdc2eeca20532 P bbf99dc978e44d32a87bf1da6d371356: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:20.738408 2019 tablet_bootstrap.cc:492] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3: No bootstrap required, opened a new log
I20260924 00:19:20.738466 2019 ts_tablet_manager.cc:1397] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 00:19:20.738639 2019 raft_consensus.cc:348] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } }
I20260924 00:19:20.738721 2019 raft_consensus.cc:374] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:20.738745 2019 raft_consensus.cc:729] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99391323ce9d4936bb69ea0e331929c3, State: Initialized, Role: FOLLOWER
I20260924 00:19:20.738804 2019 consensus_queue.cc:260] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } }
I20260924 00:19:20.738919 2019 ts_tablet_manager.cc:1428] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:20.740257 2014 tablet_bootstrap.cc:492] T ab2c4866567c4ffb9f0cdc2eeca20532 P 9fe527fb953047a8970207a900a36a3e: No bootstrap required, opened a new log
I20260924 00:19:20.740319 2014 ts_tablet_manager.cc:1397] T ab2c4866567c4ffb9f0cdc2eeca20532 P 9fe527fb953047a8970207a900a36a3e: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260924 00:19:20.740481 2014 raft_consensus.cc:348] T ab2c4866567c4ffb9f0cdc2eeca20532 P 9fe527fb953047a8970207a900a36a3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } }
I20260924 00:19:20.740561 2014 raft_consensus.cc:374] T ab2c4866567c4ffb9f0cdc2eeca20532 P 9fe527fb953047a8970207a900a36a3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:20.740583 2014 raft_consensus.cc:729] T ab2c4866567c4ffb9f0cdc2eeca20532 P 9fe527fb953047a8970207a900a36a3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe527fb953047a8970207a900a36a3e, State: Initialized, Role: FOLLOWER
I20260924 00:19:20.740639 2014 consensus_queue.cc:260] T ab2c4866567c4ffb9f0cdc2eeca20532 P 9fe527fb953047a8970207a900a36a3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } }
I20260924 00:19:20.740732 2014 ts_tablet_manager.cc:1428] T ab2c4866567c4ffb9f0cdc2eeca20532 P 9fe527fb953047a8970207a900a36a3e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:20.741400 2013 tablet_bootstrap.cc:492] T ab2c4866567c4ffb9f0cdc2eeca20532 P bbf99dc978e44d32a87bf1da6d371356: No bootstrap required, opened a new log
I20260924 00:19:20.741456 2013 ts_tablet_manager.cc:1397] T ab2c4866567c4ffb9f0cdc2eeca20532 P bbf99dc978e44d32a87bf1da6d371356: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 00:19:20.741617 2013 raft_consensus.cc:348] T ab2c4866567c4ffb9f0cdc2eeca20532 P bbf99dc978e44d32a87bf1da6d371356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } }
I20260924 00:19:20.741709 2013 raft_consensus.cc:374] T ab2c4866567c4ffb9f0cdc2eeca20532 P bbf99dc978e44d32a87bf1da6d371356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:20.741734 2013 raft_consensus.cc:729] T ab2c4866567c4ffb9f0cdc2eeca20532 P bbf99dc978e44d32a87bf1da6d371356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf99dc978e44d32a87bf1da6d371356, State: Initialized, Role: FOLLOWER
I20260924 00:19:20.741796 2013 consensus_queue.cc:260] T ab2c4866567c4ffb9f0cdc2eeca20532 P bbf99dc978e44d32a87bf1da6d371356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } }
I20260924 00:19:20.741889 2013 ts_tablet_manager.cc:1428] T ab2c4866567c4ffb9f0cdc2eeca20532 P bbf99dc978e44d32a87bf1da6d371356: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:20.742699 2035 ts_tablet_manager.cc:1905] T 95437b9d6274489eb52090591831ed6b P 9fe527fb953047a8970207a900a36a3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:20.744439 2037 tablet_replica.cc:330] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356: stopping tablet replica
I20260924 00:19:20.744575 2036 ts_tablet_manager.cc:1905] T 95437b9d6274489eb52090591831ed6b P 99391323ce9d4936bb69ea0e331929c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:20.744647 2037 raft_consensus.cc:2229] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:19:20.744760 2037 raft_consensus.cc:2258] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:20.744870 2035 ts_tablet_manager.cc:1918] T 95437b9d6274489eb52090591831ed6b P 9fe527fb953047a8970207a900a36a3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:19:20.745119 2035 log.cc:1196] T 95437b9d6274489eb52090591831ed6b P 9fe527fb953047a8970207a900a36a3e: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/95437b9d6274489eb52090591831ed6b
I20260924 00:19:20.745290 2037 ts_tablet_manager.cc:1905] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:20.745633 2035 ts_tablet_manager.cc:1939] T 95437b9d6274489eb52090591831ed6b P 9fe527fb953047a8970207a900a36a3e: Deleting consensus metadata
I20260924 00:19:20.746806 1491 catalog_manager.cc:4841] TS 9fe527fb953047a8970207a900a36a3e (127.0.98.3:38631): tablet 95437b9d6274489eb52090591831ed6b (table test-table [id=cbc8869470534aefad07ddbd40bb71b9]) successfully deleted
I20260924 00:19:20.746909 2036 ts_tablet_manager.cc:1918] T 95437b9d6274489eb52090591831ed6b P 99391323ce9d4936bb69ea0e331929c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:19:20.747066 2037 ts_tablet_manager.cc:1918] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:19:20.747143 2037 log.cc:1196] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-1/wal/wals/95437b9d6274489eb52090591831ed6b
I20260924 00:19:20.747102 2036 log.cc:1196] T 95437b9d6274489eb52090591831ed6b P 99391323ce9d4936bb69ea0e331929c3: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-0/wal/wals/95437b9d6274489eb52090591831ed6b
I20260924 00:19:20.747388 2037 ts_tablet_manager.cc:1939] T 95437b9d6274489eb52090591831ed6b P bbf99dc978e44d32a87bf1da6d371356: Deleting consensus metadata
I20260924 00:19:20.747624 2036 ts_tablet_manager.cc:1939] T 95437b9d6274489eb52090591831ed6b P 99391323ce9d4936bb69ea0e331929c3: Deleting consensus metadata
I20260924 00:19:20.747787 1494 catalog_manager.cc:4841] TS bbf99dc978e44d32a87bf1da6d371356 (127.0.98.2:45617): tablet 95437b9d6274489eb52090591831ed6b (table test-table [id=cbc8869470534aefad07ddbd40bb71b9]) successfully deleted
I20260924 00:19:20.748158 1494 catalog_manager.cc:4841] TS 99391323ce9d4936bb69ea0e331929c3 (127.0.98.1:44767): tablet 95437b9d6274489eb52090591831ed6b (table test-table [id=cbc8869470534aefad07ddbd40bb71b9]) successfully deleted
I20260924 00:19:20.806551 2021 raft_consensus.cc:482] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:19:20.806675 2021 raft_consensus.cc:504] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } }
I20260924 00:19:20.807010 2021 leader_election.cc:288] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf99dc978e44d32a87bf1da6d371356 (127.0.98.2:45617), 9fe527fb953047a8970207a900a36a3e (127.0.98.3:38631)
I20260924 00:19:20.810622 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.807224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45617 (local address 127.0.98.1:47041)
0924 00:19:20.807514 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:20.807524 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:20.807604 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:20.807766 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:20.807766 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:20.807768 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:20.807869 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.807872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.808687 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.808688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.809662 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.809665 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.810310 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.810311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.810411 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.810415 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:19:20.810481 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 00:19:20.810524 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":166,"thread_start_us":73,"threads_started":1}
I20260924 00:19:20.810626 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.807400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47041 (local address 127.0.98.2:45617)
0924 00:19:20.807465 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.807467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.807589 (+ 122us) server_negotiation.cc:407] Connection header received
0924 00:19:20.807661 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.807662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.807672 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.807759 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:20.807940 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.808603 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.809757 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.810220 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.810289 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.810301 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.810498 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.810543 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 00:19:20.810904 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.807347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38201 (local address 127.0.98.3:38631)
0924 00:19:20.807405 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.807407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.807560 (+ 153us) server_negotiation.cc:407] Connection header received
0924 00:19:20.807654 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.807656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.807666 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.807696 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:20.807942 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.808691 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.810038 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.810540 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.810604 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.810614 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.810808 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.810841 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 00:19:20.810914 2041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.807224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38631 (local address 127.0.98.1:38201)
0924 00:19:20.807464 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:20.807477 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:20.807559 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:20.807765 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:20.807766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:20.807768 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:20.807858 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.807860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.808810 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.808811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.809944 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.809947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.810667 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.810668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.810739 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.810743 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:19:20.810799 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 00:19:20.810836 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":1,"thread_start_us":69,"threads_started":1}
I20260924 00:19:20.811357 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2c4866567c4ffb9f0cdc2eeca20532" candidate_uuid: "99391323ce9d4936bb69ea0e331929c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf99dc978e44d32a87bf1da6d371356" is_pre_election: true
I20260924 00:19:20.811383 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2c4866567c4ffb9f0cdc2eeca20532" candidate_uuid: "99391323ce9d4936bb69ea0e331929c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe527fb953047a8970207a900a36a3e" is_pre_election: true
I20260924 00:19:20.811496 1943 raft_consensus.cc:2454] T ab2c4866567c4ffb9f0cdc2eeca20532 P 9fe527fb953047a8970207a900a36a3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99391323ce9d4936bb69ea0e331929c3 in term 0.
I20260924 00:19:20.811499 1804 raft_consensus.cc:2454] T ab2c4866567c4ffb9f0cdc2eeca20532 P bbf99dc978e44d32a87bf1da6d371356 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99391323ce9d4936bb69ea0e331929c3 in term 0.
I20260924 00:19:20.811685 1598 leader_election.cc:302] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99391323ce9d4936bb69ea0e331929c3, bbf99dc978e44d32a87bf1da6d371356; no voters:
I20260924 00:19:20.811813 2021 raft_consensus.cc:2781] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:19:20.811857 2021 raft_consensus.cc:482] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:19:20.811888 2021 raft_consensus.cc:3037] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:20.812892 2021 raft_consensus.cc:504] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } }
I20260924 00:19:20.813093 2021 leader_election.cc:288] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3 [CANDIDATE]: Term 1 election: Requested vote from peers bbf99dc978e44d32a87bf1da6d371356 (127.0.98.2:45617), 9fe527fb953047a8970207a900a36a3e (127.0.98.3:38631)
I20260924 00:19:20.813326 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2c4866567c4ffb9f0cdc2eeca20532" candidate_uuid: "99391323ce9d4936bb69ea0e331929c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf99dc978e44d32a87bf1da6d371356"
I20260924 00:19:20.813323 1943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2c4866567c4ffb9f0cdc2eeca20532" candidate_uuid: "99391323ce9d4936bb69ea0e331929c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe527fb953047a8970207a900a36a3e"
I20260924 00:19:20.813418 1943 raft_consensus.cc:3037] T ab2c4866567c4ffb9f0cdc2eeca20532 P 9fe527fb953047a8970207a900a36a3e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:20.813419 1804 raft_consensus.cc:3037] T ab2c4866567c4ffb9f0cdc2eeca20532 P bbf99dc978e44d32a87bf1da6d371356 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:20.814059 1804 raft_consensus.cc:2454] T ab2c4866567c4ffb9f0cdc2eeca20532 P bbf99dc978e44d32a87bf1da6d371356 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99391323ce9d4936bb69ea0e331929c3 in term 1.
I20260924 00:19:20.814208 1943 raft_consensus.cc:2454] T ab2c4866567c4ffb9f0cdc2eeca20532 P 9fe527fb953047a8970207a900a36a3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99391323ce9d4936bb69ea0e331929c3 in term 1.
I20260924 00:19:20.814416 1598 leader_election.cc:302] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99391323ce9d4936bb69ea0e331929c3, bbf99dc978e44d32a87bf1da6d371356; no voters:
I20260924 00:19:20.814507 2021 raft_consensus.cc:2781] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:20.814671 2021 raft_consensus.cc:686] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3 [term 1 LEADER]: Becoming Leader. State: Replica: 99391323ce9d4936bb69ea0e331929c3, State: Running, Role: LEADER
I20260924 00:19:20.814749 2021 consensus_queue.cc:237] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } }
I20260924 00:19:20.815521 1506 catalog_manager.cc:5495] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99391323ce9d4936bb69ea0e331929c3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "99391323ce9d4936bb69ea0e331929c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } health_report { overall_health: UNKNOWN } } }
Sep 24 00:19:20 dist-test-slave-777x krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:20.864483 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.857051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38677 (local address 127.0.0.1:36002)
0924 00:19:20.857198 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:20.857208 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:20.857308 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:20.857454 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:20.857455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:20.857820 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:20.857868 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.857870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.858544 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.858545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.859090 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.859091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.859449 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.859450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.859498 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.860062 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:20.860067 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:20.861668 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:20.863405 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:20.863414 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:20.863417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:20.863653 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:20.863866 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:20.863867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:20.863868 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:20.863925 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:20.864202 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:20.864209 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:20.864264 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 00:19:20.864312 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 00:19:20.864400 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260924 00:19:20.864858 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.857112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36002 (local address 127.0.98.62:38677)
0924 00:19:20.857173 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.857176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.857288 (+ 112us) server_negotiation.cc:407] Connection header received
0924 00:19:20.857369 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.857370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.857380 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.857450 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:20.857977 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.858464 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.859183 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.859367 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.859449 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.861772 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:20.861773 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:20.861774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:20.861782 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:20.863303 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:20.863735 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:20.863736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:20.863737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:20.863800 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:20.863996 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:20.863997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:20.863997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:20.864115 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:20.864215 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.864720 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.864786 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 00:19:20.867009 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36002:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 00:19:20.867220 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:19:20.868654 1644 tablet_service.cc:1467] Processing CreateTablet for tablet e9859b956edb4b988279090b2498c689 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1a6839cc038f45c4b17ab9178bd0867b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:19:20.868705 1924 tablet_service.cc:1467] Processing CreateTablet for tablet e9859b956edb4b988279090b2498c689 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1a6839cc038f45c4b17ab9178bd0867b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:19:20.868834 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9859b956edb4b988279090b2498c689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:20.868834 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9859b956edb4b988279090b2498c689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:20.870191 1784 tablet_service.cc:1467] Processing CreateTablet for tablet e9859b956edb4b988279090b2498c689 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1a6839cc038f45c4b17ab9178bd0867b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:19:20.870364 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9859b956edb4b988279090b2498c689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:20.870388 2019 tablet_bootstrap.cc:492] T e9859b956edb4b988279090b2498c689 P 99391323ce9d4936bb69ea0e331929c3: Bootstrap starting.
I20260924 00:19:20.870388 2014 tablet_bootstrap.cc:492] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e: Bootstrap starting.
I20260924 00:19:20.870755 2019 tablet_bootstrap.cc:654] T e9859b956edb4b988279090b2498c689 P 99391323ce9d4936bb69ea0e331929c3: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:20.870817 2014 tablet_bootstrap.cc:654] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:20.871439 2019 tablet_bootstrap.cc:492] T e9859b956edb4b988279090b2498c689 P 99391323ce9d4936bb69ea0e331929c3: No bootstrap required, opened a new log
I20260924 00:19:20.871490 2019 ts_tablet_manager.cc:1397] T e9859b956edb4b988279090b2498c689 P 99391323ce9d4936bb69ea0e331929c3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:20.871608 2014 tablet_bootstrap.cc:492] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e: No bootstrap required, opened a new log
I20260924 00:19:20.871641 2014 ts_tablet_manager.cc:1397] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:20.871632 2019 raft_consensus.cc:348] T e9859b956edb4b988279090b2498c689 P 99391323ce9d4936bb69ea0e331929c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } }
I20260924 00:19:20.871709 2019 raft_consensus.cc:374] T e9859b956edb4b988279090b2498c689 P 99391323ce9d4936bb69ea0e331929c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:20.871732 2019 raft_consensus.cc:729] T e9859b956edb4b988279090b2498c689 P 99391323ce9d4936bb69ea0e331929c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99391323ce9d4936bb69ea0e331929c3, State: Initialized, Role: FOLLOWER
I20260924 00:19:20.871781 2019 consensus_queue.cc:260] T e9859b956edb4b988279090b2498c689 P 99391323ce9d4936bb69ea0e331929c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } }
I20260924 00:19:20.871768 2014 raft_consensus.cc:348] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } }
I20260924 00:19:20.871836 2014 raft_consensus.cc:374] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:20.871857 2014 raft_consensus.cc:729] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe527fb953047a8970207a900a36a3e, State: Initialized, Role: FOLLOWER
I20260924 00:19:20.871886 2019 ts_tablet_manager.cc:1428] T e9859b956edb4b988279090b2498c689 P 99391323ce9d4936bb69ea0e331929c3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:20.871899 2014 consensus_queue.cc:260] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } }
I20260924 00:19:20.871963 2018 raft_consensus.cc:482] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:19:20.872004 2018 raft_consensus.cc:504] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } }
I20260924 00:19:20.872048 2014 ts_tablet_manager.cc:1428] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 00:19:20.872006 2021 tablet_replica.cc:439] T e9859b956edb4b988279090b2498c689 P 99391323ce9d4936bb69ea0e331929c3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } } }
I20260924 00:19:20.872079 2013 tablet_bootstrap.cc:492] T e9859b956edb4b988279090b2498c689 P bbf99dc978e44d32a87bf1da6d371356: Bootstrap starting.
I20260924 00:19:20.872092 2021 tablet_replica.cc:442] T e9859b956edb4b988279090b2498c689 P 99391323ce9d4936bb69ea0e331929c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:19:20.872310 2018 leader_election.cc:288] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99391323ce9d4936bb69ea0e331929c3 (127.0.98.1:44767), bbf99dc978e44d32a87bf1da6d371356 (127.0.98.2:45617)
I20260924 00:19:20.872417 2013 tablet_bootstrap.cc:654] T e9859b956edb4b988279090b2498c689 P bbf99dc978e44d32a87bf1da6d371356: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:20.872413 2018 tablet_replica.cc:439] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } } }
I20260924 00:19:20.872493 2018 tablet_replica.cc:442] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:19:20.873116 2013 tablet_bootstrap.cc:492] T e9859b956edb4b988279090b2498c689 P bbf99dc978e44d32a87bf1da6d371356: No bootstrap required, opened a new log
I20260924 00:19:20.873167 2013 ts_tablet_manager.cc:1397] T e9859b956edb4b988279090b2498c689 P bbf99dc978e44d32a87bf1da6d371356: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:20.873271 2013 raft_consensus.cc:348] T e9859b956edb4b988279090b2498c689 P bbf99dc978e44d32a87bf1da6d371356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } }
I20260924 00:19:20.873328 2013 raft_consensus.cc:374] T e9859b956edb4b988279090b2498c689 P bbf99dc978e44d32a87bf1da6d371356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:20.873346 2013 raft_consensus.cc:729] T e9859b956edb4b988279090b2498c689 P bbf99dc978e44d32a87bf1da6d371356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf99dc978e44d32a87bf1da6d371356, State: Initialized, Role: FOLLOWER
I20260924 00:19:20.873384 2013 consensus_queue.cc:260] T e9859b956edb4b988279090b2498c689 P bbf99dc978e44d32a87bf1da6d371356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } }
I20260924 00:19:20.873459 2013 ts_tablet_manager.cc:1428] T e9859b956edb4b988279090b2498c689 P bbf99dc978e44d32a87bf1da6d371356: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:20.873558 2023 tablet_replica.cc:439] T e9859b956edb4b988279090b2498c689 P bbf99dc978e44d32a87bf1da6d371356: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } } }
I20260924 00:19:20.873611 2023 tablet_replica.cc:442] T e9859b956edb4b988279090b2498c689 P bbf99dc978e44d32a87bf1da6d371356: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:19:20.875526 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.872520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44767 (local address 127.0.98.3:55299)
0924 00:19:20.872637 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:20.872651 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:20.872697 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:20.872991 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:20.872992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:20.872994 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:20.873063 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.873065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.873941 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.873941 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.874684 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.874687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.875279 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.875280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.875364 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.875368 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:19:20.875421 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 00:19:20.875461 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 00:19:20.875730 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.872811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55299 (local address 127.0.98.1:44767)
0924 00:19:20.872849 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.872851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.872858 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:19:20.872881 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.872881 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.872890 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.872942 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:20.873141 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.873861 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.874786 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.875190 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.875282 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.875293 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.875628 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.875667 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:19:20.876232 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.872783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41363 (local address 127.0.98.2:45617)
0924 00:19:20.872821 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.872823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.872832 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:19:20.872906 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.872907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.872918 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.872948 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:20.873525 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.874184 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.875098 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.875562 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.875651 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.875664 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.876108 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.876146 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 00:19:20.876247 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.872624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45617 (local address 127.0.98.3:41363)
0924 00:19:20.872756 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:20.872766 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:20.872825 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:20.873011 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:20.873012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:20.873014 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:20.873434 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.873437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.874279 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.874279 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.875004 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.875006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.875838 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.875839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.875917 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.875921 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:19:20.876097 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 00:19:20.876147 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260924 00:19:20.876454 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9859b956edb4b988279090b2498c689" candidate_uuid: "9fe527fb953047a8970207a900a36a3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf99dc978e44d32a87bf1da6d371356" is_pre_election: true
I20260924 00:19:20.876555 1804 raft_consensus.cc:2454] T e9859b956edb4b988279090b2498c689 P bbf99dc978e44d32a87bf1da6d371356 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fe527fb953047a8970207a900a36a3e in term 0.
I20260924 00:19:20.876760 1878 leader_election.cc:302] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fe527fb953047a8970207a900a36a3e, bbf99dc978e44d32a87bf1da6d371356; no voters:
I20260924 00:19:20.876893 2053 raft_consensus.cc:2781] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:19:20.876943 2053 raft_consensus.cc:482] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:19:20.876986 2053 raft_consensus.cc:3037] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:20.877688 2053 raft_consensus.cc:504] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } }
I20260924 00:19:20.877828 2053 leader_election.cc:288] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e [CANDIDATE]: Term 1 election: Requested vote from peers 99391323ce9d4936bb69ea0e331929c3 (127.0.98.1:44767), bbf99dc978e44d32a87bf1da6d371356 (127.0.98.2:45617)
I20260924 00:19:20.877982 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9859b956edb4b988279090b2498c689" candidate_uuid: "9fe527fb953047a8970207a900a36a3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf99dc978e44d32a87bf1da6d371356"
I20260924 00:19:20.878058 1804 raft_consensus.cc:3037] T e9859b956edb4b988279090b2498c689 P bbf99dc978e44d32a87bf1da6d371356 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:20.878504 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9859b956edb4b988279090b2498c689" candidate_uuid: "9fe527fb953047a8970207a900a36a3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99391323ce9d4936bb69ea0e331929c3" is_pre_election: true
I20260924 00:19:20.878605 1664 raft_consensus.cc:2454] T e9859b956edb4b988279090b2498c689 P 99391323ce9d4936bb69ea0e331929c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fe527fb953047a8970207a900a36a3e in term 0.
I20260924 00:19:20.878557 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9859b956edb4b988279090b2498c689" candidate_uuid: "9fe527fb953047a8970207a900a36a3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99391323ce9d4936bb69ea0e331929c3"
I20260924 00:19:20.878724 1663 raft_consensus.cc:3037] T e9859b956edb4b988279090b2498c689 P 99391323ce9d4936bb69ea0e331929c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:20.878785 1804 raft_consensus.cc:2454] T e9859b956edb4b988279090b2498c689 P bbf99dc978e44d32a87bf1da6d371356 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe527fb953047a8970207a900a36a3e in term 1.
I20260924 00:19:20.878912 1878 leader_election.cc:302] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fe527fb953047a8970207a900a36a3e, bbf99dc978e44d32a87bf1da6d371356; no voters:
I20260924 00:19:20.878995 2053 raft_consensus.cc:2781] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:20.879042 2053 raft_consensus.cc:686] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e [term 1 LEADER]: Becoming Leader. State: Replica: 9fe527fb953047a8970207a900a36a3e, State: Running, Role: LEADER
I20260924 00:19:20.879101 2053 consensus_queue.cc:237] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } }
I20260924 00:19:20.879498 2018 tablet_replica.cc:439] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e: TxnStatusTablet state changed. Reason: New leader 9fe527fb953047a8970207a900a36a3e. Latest consensus state: current_term: 1 leader_uuid: "9fe527fb953047a8970207a900a36a3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } } }
I20260924 00:19:20.879586 2018 tablet_replica.cc:442] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:19:20.879603 1663 raft_consensus.cc:2454] T e9859b956edb4b988279090b2498c689 P 99391323ce9d4936bb69ea0e331929c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe527fb953047a8970207a900a36a3e in term 1.
I20260924 00:19:20.879793 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 00:19:20.879885 1506 catalog_manager.cc:5495] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fe527fb953047a8970207a900a36a3e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9fe527fb953047a8970207a900a36a3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } health_report { overall_health: UNKNOWN } } }
I20260924 00:19:20.880051 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 00:19:20.888727 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.885731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38631 (local address 127.0.0.1:51566)
0924 00:19:20.885839 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:20.885852 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:20.885905 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:20.886139 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:20.886140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:20.886142 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:19:20.886211 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.886213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.886851 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.886852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.887421 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.887423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.888075 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.888076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.888153 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.888248 (+ 95us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:19:20.888541 (+ 293us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:19:20.888550 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 00:19:20.888607 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 00:19:20.888660 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 00:19:20.888720 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.885783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51566 (local address 127.0.98.3:38631)
0924 00:19:20.885848 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.885851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.885885 (+ 34us) server_negotiation.cc:407] Connection header received
0924 00:19:20.886024 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.886025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.886051 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.886116 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:20.886277 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.886756 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.887521 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.887989 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.888076 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.888354 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:20.888473 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:20.888556 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.888613 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.888635 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 00:19:20.889842 1663 raft_consensus.cc:1261] T e9859b956edb4b988279090b2498c689 P 99391323ce9d4936bb69ea0e331929c3 [term 1 FOLLOWER]: Refusing update from remote peer 9fe527fb953047a8970207a900a36a3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:20.889889 1804 raft_consensus.cc:1261] T e9859b956edb4b988279090b2498c689 P bbf99dc978e44d32a87bf1da6d371356 [term 1 FOLLOWER]: Refusing update from remote peer 9fe527fb953047a8970207a900a36a3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:20.890165 2018 consensus_queue.cc:1035] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:20.890311 2053 consensus_queue.cc:1035] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:20.890847 2021 tablet_replica.cc:439] T e9859b956edb4b988279090b2498c689 P 99391323ce9d4936bb69ea0e331929c3: TxnStatusTablet state changed. Reason: New leader 9fe527fb953047a8970207a900a36a3e. Latest consensus state: current_term: 1 leader_uuid: "9fe527fb953047a8970207a900a36a3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } } }
I20260924 00:19:20.890944 2021 tablet_replica.cc:442] T e9859b956edb4b988279090b2498c689 P 99391323ce9d4936bb69ea0e331929c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:19:20.891085 2023 tablet_replica.cc:439] T e9859b956edb4b988279090b2498c689 P bbf99dc978e44d32a87bf1da6d371356: TxnStatusTablet state changed. Reason: New leader 9fe527fb953047a8970207a900a36a3e. Latest consensus state: current_term: 1 leader_uuid: "9fe527fb953047a8970207a900a36a3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } } }
I20260924 00:19:20.891170 2023 tablet_replica.cc:442] T e9859b956edb4b988279090b2498c689 P bbf99dc978e44d32a87bf1da6d371356: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:19:20.891420 2053 tablet_replica.cc:439] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fe527fb953047a8970207a900a36a3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } } }
I20260924 00:19:20.891494 2053 tablet_replica.cc:442] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:19:20.891978 2021 tablet_replica.cc:439] T e9859b956edb4b988279090b2498c689 P 99391323ce9d4936bb69ea0e331929c3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fe527fb953047a8970207a900a36a3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } } }
I20260924 00:19:20.892063 2021 tablet_replica.cc:442] T e9859b956edb4b988279090b2498c689 P 99391323ce9d4936bb69ea0e331929c3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:19:20.893055 2053 tablet_replica.cc:439] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9fe527fb953047a8970207a900a36a3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } } }
I20260924 00:19:20.893158 2053 tablet_replica.cc:442] T e9859b956edb4b988279090b2498c689 P 9fe527fb953047a8970207a900a36a3e: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:19:20.893611 2023 tablet_replica.cc:439] T e9859b956edb4b988279090b2498c689 P bbf99dc978e44d32a87bf1da6d371356: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9fe527fb953047a8970207a900a36a3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99391323ce9d4936bb69ea0e331929c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44767 } } peers { permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 } } peers { permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 } } }
I20260924 00:19:20.893699 2023 tablet_replica.cc:442] T e9859b956edb4b988279090b2498c689 P bbf99dc978e44d32a87bf1da6d371356: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:19:20.895782 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.893087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49228 (local address 127.0.98.1:44767)
0924 00:19:20.893127 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.893129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.893138 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:19:20.893239 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.893240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.893260 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.893319 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:20.893484 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.894261 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.894968 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.895275 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.895345 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.895454 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:20.895583 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:20.895656 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.895674 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.895692 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:19:20.901490 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.898498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50473 (local address 127.0.98.3:38631)
0924 00:19:20.898557 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.898560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.898823 (+ 263us) server_negotiation.cc:407] Connection header received
0924 00:19:20.898858 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.898859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.898868 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.898932 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:20.899084 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.899576 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.900761 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.900977 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.901045 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.901234 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:20.901260 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:20.901325 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.901404 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.901431 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 00:19:20.901553 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.898445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38631 (local address 127.0.98.1:50473)
0924 00:19:20.898661 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:20.898673 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:20.898738 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:20.898948 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:20.898949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:20.898951 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:19:20.899016 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.899017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.899651 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.899652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.900671 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.900674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.901060 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.901061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.901132 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.901150 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:19:20.901345 (+ 195us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:19:20.901347 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:19:20.901401 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 00:19:20.901443 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":115,"threads_started":1}
I20260924 00:19:20.904368 1804 raft_consensus.cc:1261] T ab2c4866567c4ffb9f0cdc2eeca20532 P bbf99dc978e44d32a87bf1da6d371356 [term 1 FOLLOWER]: Refusing update from remote peer 99391323ce9d4936bb69ea0e331929c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:20.904363 1943 raft_consensus.cc:1261] T ab2c4866567c4ffb9f0cdc2eeca20532 P 9fe527fb953047a8970207a900a36a3e [term 1 FOLLOWER]: Refusing update from remote peer 99391323ce9d4936bb69ea0e331929c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:20.904815 2043 consensus_queue.cc:1035] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe527fb953047a8970207a900a36a3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:20.904965 2021 consensus_queue.cc:1035] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf99dc978e44d32a87bf1da6d371356" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:20.914232 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.911140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39153 (local address 127.0.98.1:44767)
0924 00:19:20.911184 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.911186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.911243 (+ 57us) server_negotiation.cc:407] Connection header received
0924 00:19:20.911318 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.911319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.911329 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.911381 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:20.911750 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.912300 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.913445 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.913734 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.913812 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.914007 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:20.914034 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:20.914078 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.914132 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.914149 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 00:19:20.914266 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.911074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44767 (local address 127.0.98.3:39153)
0924 00:19:20.911201 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:20.911211 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:20.911261 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:20.911402 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:20.911403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:20.911405 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:19:20.911676 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.911678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.912376 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.912377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.913314 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:20.913316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.913814 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.913816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:20.913890 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.913916 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:19:20.914079 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:19:20.914080 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 00:19:20.914130 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 00:19:20.914172 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 00:19:20.925226 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:20.925361 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:20.926071 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7d1c925c95f4013a9d782c44266993f: Sending DeleteTablet for 3 replicas of tablet ab2c4866567c4ffb9f0cdc2eeca20532
I20260924 00:19:20.926350 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet ab2c4866567c4ffb9f0cdc2eeca20532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51552
I20260924 00:19:20.926427 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet ab2c4866567c4ffb9f0cdc2eeca20532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43432
I20260924 00:19:20.926477 2035 tablet_replica.cc:330] T ab2c4866567c4ffb9f0cdc2eeca20532 P 9fe527fb953047a8970207a900a36a3e: stopping tablet replica
I20260924 00:19:20.926527 2037 tablet_replica.cc:330] T ab2c4866567c4ffb9f0cdc2eeca20532 P bbf99dc978e44d32a87bf1da6d371356: stopping tablet replica
I20260924 00:19:20.926543 2035 raft_consensus.cc:2229] T ab2c4866567c4ffb9f0cdc2eeca20532 P 9fe527fb953047a8970207a900a36a3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:20.926585 2035 raft_consensus.cc:2258] T ab2c4866567c4ffb9f0cdc2eeca20532 P 9fe527fb953047a8970207a900a36a3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:20.926578 2037 raft_consensus.cc:2229] T ab2c4866567c4ffb9f0cdc2eeca20532 P bbf99dc978e44d32a87bf1da6d371356 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:20.926625 2037 raft_consensus.cc:2258] T ab2c4866567c4ffb9f0cdc2eeca20532 P bbf99dc978e44d32a87bf1da6d371356 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:20.926805 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet ab2c4866567c4ffb9f0cdc2eeca20532 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49222
I20260924 00:19:20.926944 2035 ts_tablet_manager.cc:1905] T ab2c4866567c4ffb9f0cdc2eeca20532 P 9fe527fb953047a8970207a900a36a3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:20.927189 2037 ts_tablet_manager.cc:1905] T ab2c4866567c4ffb9f0cdc2eeca20532 P bbf99dc978e44d32a87bf1da6d371356: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:20.928682 2035 ts_tablet_manager.cc:1918] T ab2c4866567c4ffb9f0cdc2eeca20532 P 9fe527fb953047a8970207a900a36a3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:19:20.928682 2037 ts_tablet_manager.cc:1918] T ab2c4866567c4ffb9f0cdc2eeca20532 P bbf99dc978e44d32a87bf1da6d371356: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:19:20.928751 2037 log.cc:1196] T ab2c4866567c4ffb9f0cdc2eeca20532 P bbf99dc978e44d32a87bf1da6d371356: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-1/wal/wals/ab2c4866567c4ffb9f0cdc2eeca20532
I20260924 00:19:20.928751 2035 log.cc:1196] T ab2c4866567c4ffb9f0cdc2eeca20532 P 9fe527fb953047a8970207a900a36a3e: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/ab2c4866567c4ffb9f0cdc2eeca20532
I20260924 00:19:20.928989 2037 ts_tablet_manager.cc:1939] T ab2c4866567c4ffb9f0cdc2eeca20532 P bbf99dc978e44d32a87bf1da6d371356: Deleting consensus metadata
I20260924 00:19:20.928989 2035 ts_tablet_manager.cc:1939] T ab2c4866567c4ffb9f0cdc2eeca20532 P 9fe527fb953047a8970207a900a36a3e: Deleting consensus metadata
I20260924 00:19:20.929127 2036 tablet_replica.cc:330] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3: stopping tablet replica
I20260924 00:19:20.929194 2036 raft_consensus.cc:2229] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:19:20.929343 2036 raft_consensus.cc:2258] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:20.929513 1491 catalog_manager.cc:4841] TS 9fe527fb953047a8970207a900a36a3e (127.0.98.3:38631): tablet ab2c4866567c4ffb9f0cdc2eeca20532 (table test-table [id=0c23bdef146b4d99a2c550305e448002]) successfully deleted
I20260924 00:19:20.929679 1494 catalog_manager.cc:4841] TS bbf99dc978e44d32a87bf1da6d371356 (127.0.98.2:45617): tablet ab2c4866567c4ffb9f0cdc2eeca20532 (table test-table [id=0c23bdef146b4d99a2c550305e448002]) successfully deleted
I20260924 00:19:20.929883 2036 ts_tablet_manager.cc:1905] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:20.931360 2036 ts_tablet_manager.cc:1918] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:19:20.931420 2036 log.cc:1196] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790209154318147-392-0/minicluster-data/ts-0/wal/wals/ab2c4866567c4ffb9f0cdc2eeca20532
I20260924 00:19:20.931622 2036 ts_tablet_manager.cc:1939] T ab2c4866567c4ffb9f0cdc2eeca20532 P 99391323ce9d4936bb69ea0e331929c3: Deleting consensus metadata
I20260924 00:19:20.932090 1494 catalog_manager.cc:4841] TS 99391323ce9d4936bb69ea0e331929c3 (127.0.98.1:44767): tablet ab2c4866567c4ffb9f0cdc2eeca20532 (table test-table [id=0c23bdef146b4d99a2c550305e448002]) successfully deleted
Sep 24 00:19:20 dist-test-slave-777x krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209160, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:19:20.948213 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.928509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49232 (local address 127.0.98.1:44767)
0924 00:19:20.928555 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.928558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.936384 (+ 7826us) server_negotiation.cc:407] Connection header received
0924 00:19:20.936485 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.936487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.936501 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.936581 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:20.937469 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.938334 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.939738 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.940049 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.940125 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.944140 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:20.944151 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:20.944152 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:20.944160 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:20.946262 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:20.946693 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:20.946697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:20.946698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:20.946768 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:20.947012 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:20.947013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:20.947014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:20.947213 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:20.947370 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.947971 (+ 601us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.948057 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 00:19:20.948439 1698 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49232
I20260924 00:19:20.957603 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:20.949485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.0.98.62:38677)
0924 00:19:20.949532 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:20.949534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:20.949735 (+ 201us) server_negotiation.cc:407] Connection header received
0924 00:19:20.949790 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:20.949792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:20.949804 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:20.949865 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:20.950564 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.951352 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.952544 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:20.952853 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:20.952925 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:20.954494 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:20.954496 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:20.954496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:20.954504 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:20.956302 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:20.956775 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:20.956777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:20.956777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:20.956837 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:20.957054 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:20.957055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:20.957055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:20.957183 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:20.957276 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:20.957428 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 00:19:20.957504 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 00:19:20.957924 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36006
I20260924 00:19:20.958716 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 1579
I20260924 00:19:20.965402 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 1719
I20260924 00:19:20.972942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 1859
I20260924 00:19:20.978085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 1475
2026-09-24T00:19:20Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2460 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:19:21 dist-test-slave-777x krb5kdc[2078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:19:21 dist-test-slave-777x krb5kdc[2078](info): set up 2 sockets
Sep 24 00:19:21 dist-test-slave-777x krb5kdc[2078](info): commencing operation
krb5kdc: starting...
W20260924 00:19:22.048455 392 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:19:22 dist-test-slave-777x krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:19:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:19:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:19:22.176000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45675
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:22.256783 2091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:22.257076 2091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:22.257119 2091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:22.258371 2091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:19:22.258460 2091 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 00:19:22.258483 2091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:22.258508 2091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:22.258536 2091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:19:22.258561 2091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:19:22.260805 2091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45675
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:22.261128 2091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:22.261386 2091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:22.266531 2096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:22.266566 2097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:22.266531 2099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:22.266970 2091 server_base.cc:1034] running on GCE node
I20260924 00:19:22.267221 2091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:22.267488 2091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:22.268642 2091 hybrid_clock.cc:648] HybridClock initialized: now 1790209162268631 us; error 34 us; skew 500 ppm
Sep 24 00:19:22 dist-test-slave-777x krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:22.271554 2091 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:19:22.272074 2091 webserver.cc:460] Webserver started at http://127.0.98.62:35635/ using document root <none> and password file <none>
I20260924 00:19:22.272348 2091 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:22.272418 2091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:22.272548 2091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:22.273620 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "59eaae68dcf8488681c14e44fdf65dd6"
format_stamp: "Formatted at 2026-09-24 00:19:22 on dist-test-slave-777x"
server_key: "d2f8db495d807cfec49b1fcbe847fa31"
server_key_iv: "3956f343b4aa7e9254930eb27ac3018c"
server_key_version: "encryptionkey@0"
I20260924 00:19:22.273994 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "59eaae68dcf8488681c14e44fdf65dd6"
format_stamp: "Formatted at 2026-09-24 00:19:22 on dist-test-slave-777x"
server_key: "d2f8db495d807cfec49b1fcbe847fa31"
server_key_iv: "3956f343b4aa7e9254930eb27ac3018c"
server_key_version: "encryptionkey@0"
I20260924 00:19:22.275377 2091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:19:22.276268 2106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:22.276561 2091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:22.276666 2091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "59eaae68dcf8488681c14e44fdf65dd6"
format_stamp: "Formatted at 2026-09-24 00:19:22 on dist-test-slave-777x"
server_key: "d2f8db495d807cfec49b1fcbe847fa31"
server_key_iv: "3956f343b4aa7e9254930eb27ac3018c"
server_key_version: "encryptionkey@0"
I20260924 00:19:22.276754 2091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:22.291479 2091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:22.294042 2091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:22.294214 2091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:22.299726 2091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45675
I20260924 00:19:22.299760 2168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45675 every 8 connection(s)
I20260924 00:19:22.300159 2091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:19:22.300976 2169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:22.301035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 2091
I20260924 00:19:22.301119 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:19:22.301297 392 external_mini_cluster.cc:1428] Setting key f8d2f16377aa56d4eeb135e1c26dd01b
I20260924 00:19:22.303938 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6: Bootstrap starting.
I20260924 00:19:22.304656 2169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:22.304940 2169 log.cc:824] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:22.305625 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6: No bootstrap required, opened a new log
I20260924 00:19:22.307686 2169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59eaae68dcf8488681c14e44fdf65dd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45675 } }
I20260924 00:19:22.307884 2169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:22.307926 2169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59eaae68dcf8488681c14e44fdf65dd6, State: Initialized, Role: FOLLOWER
I20260924 00:19:22.308082 2169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59eaae68dcf8488681c14e44fdf65dd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45675 } }
I20260924 00:19:22.308197 2169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 24 00:19:22 dist-test-slave-777x krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:22.308233 2169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:19:22.308290 2169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:22.309026 2169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59eaae68dcf8488681c14e44fdf65dd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45675 } }
I20260924 00:19:22.309156 2169 leader_election.cc:302] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59eaae68dcf8488681c14e44fdf65dd6; no voters:
I20260924 00:19:22.309366 2169 leader_election.cc:288] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:19:22.309490 2174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:22.309669 2174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6 [term 1 LEADER]: Becoming Leader. State: Replica: 59eaae68dcf8488681c14e44fdf65dd6, State: Running, Role: LEADER
I20260924 00:19:22.309677 2169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:19:22.310050 2174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59eaae68dcf8488681c14e44fdf65dd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45675 } }
I20260924 00:19:22.310511 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59eaae68dcf8488681c14e44fdf65dd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59eaae68dcf8488681c14e44fdf65dd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45675 } } }
I20260924 00:19:22.310519 2176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59eaae68dcf8488681c14e44fdf65dd6. Latest consensus state: current_term: 1 leader_uuid: "59eaae68dcf8488681c14e44fdf65dd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59eaae68dcf8488681c14e44fdf65dd6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45675 } } }
I20260924 00:19:22.310619 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:22.310624 2176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:22.311234 2184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:19:22.312433 2184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:19:22.313040 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.301852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58048 (local address 127.0.98.62:45675)
0924 00:19:22.302051 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:22.302066 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:22.302093 (+ 27us) server_negotiation.cc:407] Connection header received
0924 00:19:22.302556 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:22.302563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:22.302616 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:22.302771 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:22.303460 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.304527 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.305424 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.305898 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.306018 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.308790 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:22.308796 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:22.308797 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:22.308812 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:22.311073 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:22.311478 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:22.311481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:22.311481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:22.311593 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:22.311840 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:22.311841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:22.311842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:22.312242 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:22.312395 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:22.312688 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 00:19:22.312833 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":78,"threads_started":1}
I20260924 00:19:22.317051 2184 catalog_manager.cc:1347] Generated new cluster ID: b96908e7112f4229bc36ae6542f24c80
I20260924 00:19:22.317130 2184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:19:22.330159 2184 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:19:22.330760 2184 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:19:22.337428 2184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6: Generated new TSK 0
I20260924 00:19:22.337673 2184 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:19:22.375396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45675
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:22.463116 2195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:22.463310 2195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:22.463336 2195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:22.464573 2195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:22.464617 2195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:22.464638 2195 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 00:19:22.464650 2195 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:19:22.464668 2195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:19:22.466141 2195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:22.466385 2195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:22.466614 2195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:22.471612 2203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:22.471597 2200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:22.471597 2201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:22.471661 2195 server_base.cc:1034] running on GCE node
I20260924 00:19:22.472035 2195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:22.472347 2195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:22.473531 2195 hybrid_clock.cc:648] HybridClock initialized: now 1790209162473512 us; error 35 us; skew 500 ppm
Sep 24 00:19:22 dist-test-slave-777x krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:22.475915 2195 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:19:22.476435 2195 webserver.cc:460] Webserver started at http://127.0.98.1:42153/ using document root <none> and password file <none>
I20260924 00:19:22.476671 2195 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:22.476740 2195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:22.476893 2195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:22.478005 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "86b78a425f984265aa22d4a1917f23aa"
format_stamp: "Formatted at 2026-09-24 00:19:22 on dist-test-slave-777x"
server_key: "87c1f877a8dc6b77b06654a745d2bc67"
server_key_iv: "bff2d574fef69baa62a68b61bd8c4b2b"
server_key_version: "encryptionkey@0"
I20260924 00:19:22.478374 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "86b78a425f984265aa22d4a1917f23aa"
format_stamp: "Formatted at 2026-09-24 00:19:22 on dist-test-slave-777x"
server_key: "87c1f877a8dc6b77b06654a745d2bc67"
server_key_iv: "bff2d574fef69baa62a68b61bd8c4b2b"
server_key_version: "encryptionkey@0"
I20260924 00:19:22.479679 2195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:19:22.480546 2210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:22.480727 2195 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:22.481052 2195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "86b78a425f984265aa22d4a1917f23aa"
format_stamp: "Formatted at 2026-09-24 00:19:22 on dist-test-slave-777x"
server_key: "87c1f877a8dc6b77b06654a745d2bc67"
server_key_iv: "bff2d574fef69baa62a68b61bd8c4b2b"
server_key_version: "encryptionkey@0"
I20260924 00:19:22.481148 2195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:22.498265 2195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:22.500782 2195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:22.500943 2195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:22.501492 2195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:22.501535 2195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:22.501560 2195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:22.501603 2195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:22.508716 2195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40309
I20260924 00:19:22.508821 2325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40309 every 8 connection(s)
I20260924 00:19:22.509200 2195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 00:19:22 dist-test-slave-777x krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:22.511193 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 2195
I20260924 00:19:22.511322 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:19:22.511539 392 external_mini_cluster.cc:1428] Setting key adebd25d82f6415d9a4c7e8d6ff8964d
I20260924 00:19:22.513440 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.502097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34993 (local address 127.0.98.62:45675)
0924 00:19:22.502177 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:22.502180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:22.502378 (+ 198us) server_negotiation.cc:407] Connection header received
0924 00:19:22.502969 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:22.502971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:22.502988 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:22.503076 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:22.504032 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.504956 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.506616 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.506963 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.507053 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.510076 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:22.510077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:22.510078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:22.510085 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:22.511900 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:22.512422 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:22.512423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:22.512424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:22.512500 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:22.512758 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:22.512759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:22.512759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:22.512884 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:22.512962 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:22.513259 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 00:19:22.513346 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 00:19:22.513698 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.502018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45675 (local address 127.0.98.1:34993)
0924 00:19:22.502240 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:22.502281 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:22.502797 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:22.503144 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:22.503165 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:22.503554 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:22.503857 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:22.503864 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.505075 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.505077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:22.506496 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:22.506499 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.507057 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.507058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:22.507189 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.507896 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:22.507907 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:22.509972 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:22.511985 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:22.511991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:22.511993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:22.512329 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:22.512571 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:22.512572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:22.512572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:22.512683 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:22.512953 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:22.512960 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:22.513121 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 00:19:22.513239 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 00:19:22.513409 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":113,"threads_started":1}
I20260924 00:19:22.514205 2326 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45675
I20260924 00:19:22.514310 2326 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:22.514506 2326 heartbeater.cc:511] Master 127.0.98.62:45675 requested a full tablet report, sending...
I20260924 00:19:22.515116 2122 ts_manager.cc:194] Registered new tserver with Master: 86b78a425f984265aa22d4a1917f23aa (127.0.98.1:40309)
I20260924 00:19:22.516099 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34993
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 00:19:22.522691 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.515390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58731 (local address 127.0.98.62:45675)
0924 00:19:22.515431 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:22.515434 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:22.516452 (+ 1018us) server_negotiation.cc:407] Connection header received
0924 00:19:22.516597 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:22.516598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:22.516609 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:22.516641 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:22.517099 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.517554 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.518738 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.518969 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.519031 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.520122 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:22.520123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:22.520124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:22.520129 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:22.521679 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:22.522050 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:22.522051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:22.522051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:22.522104 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:22.522273 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:22.522274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:22.522274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:22.522394 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:22.522447 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:22.522563 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 00:19:22.522624 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:19:22.522703 2333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.515342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45675 (local address 127.0.98.1:58731)
0924 00:19:22.516379 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:22.516392 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:22.516493 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:22.516702 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:22.516703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:22.516932 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:22.517003 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:22.517005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.517659 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.517661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:22.518642 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:22.518645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.519055 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.519056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:22.519144 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.519616 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:22.519622 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:22.520027 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:22.521760 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:22.521783 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:22.521783 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:22.521977 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:22.522168 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:22.522169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:22.522169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:22.522222 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:22.522451 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:22.522451 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:22.522487 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 00:19:22.522544 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 00:19:22.522627 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"thread_start_us":90,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:19:22.539783 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45675
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:22.626077 2335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:22.626294 2335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:22.626319 2335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:22.627519 2335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:22.627573 2335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:22.627595 2335 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 00:19:22.627607 2335 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:19:22.627625 2335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:19:22.629142 2335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:22.629395 2335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:22.629662 2335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:22.635114 2341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:22.635145 2343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:22.635145 2340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:22.635468 2335 server_base.cc:1034] running on GCE node
I20260924 00:19:22.635720 2335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:22.636009 2335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:22.637207 2335 hybrid_clock.cc:648] HybridClock initialized: now 1790209162637210 us; error 44 us; skew 500 ppm
Sep 24 00:19:22 dist-test-slave-777x krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:22.639871 2335 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:19:22.640411 2335 webserver.cc:460] Webserver started at http://127.0.98.2:35609/ using document root <none> and password file <none>
I20260924 00:19:22.640602 2335 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:22.640643 2335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:22.640746 2335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:22.641744 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "f59aa39d2fa147cc8faf18842d35958e"
format_stamp: "Formatted at 2026-09-24 00:19:22 on dist-test-slave-777x"
server_key: "f36bee00378c051b2d851c9ccc213f8d"
server_key_iv: "652e233cc70beae1d3f3e0d8abf7b177"
server_key_version: "encryptionkey@0"
I20260924 00:19:22.642061 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f59aa39d2fa147cc8faf18842d35958e"
format_stamp: "Formatted at 2026-09-24 00:19:22 on dist-test-slave-777x"
server_key: "f36bee00378c051b2d851c9ccc213f8d"
server_key_iv: "652e233cc70beae1d3f3e0d8abf7b177"
server_key_version: "encryptionkey@0"
I20260924 00:19:22.643388 2335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:19:22.644213 2350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:22.644407 2335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:22.644490 2335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "f59aa39d2fa147cc8faf18842d35958e"
format_stamp: "Formatted at 2026-09-24 00:19:22 on dist-test-slave-777x"
server_key: "f36bee00378c051b2d851c9ccc213f8d"
server_key_iv: "652e233cc70beae1d3f3e0d8abf7b177"
server_key_version: "encryptionkey@0"
I20260924 00:19:22.644550 2335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:22.655853 2335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:22.658274 2335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:22.658437 2335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:22.659077 2335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:22.659121 2335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:22.659143 2335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:22.659157 2335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 00:19:22 dist-test-slave-777x krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:22.666612 2335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32917
I20260924 00:19:22.667045 2335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:19:22.667039 2465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32917 every 8 connection(s)
I20260924 00:19:22.669387 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.659616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58043 (local address 127.0.98.62:45675)
0924 00:19:22.659693 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:22.659696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:22.659942 (+ 246us) server_negotiation.cc:407] Connection header received
0924 00:19:22.660587 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:22.660588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:22.660601 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:22.660666 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:22.661323 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.661878 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.663014 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.663251 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.663337 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.666138 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:22.666140 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:22.666140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:22.666148 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:22.667878 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:22.668406 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:22.668407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:22.668408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:22.668475 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:22.668730 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:22.668731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:22.668731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:22.668855 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:22.668929 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:22.669202 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 00:19:22.669288 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 00:19:22.669561 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.659597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45675 (local address 127.0.98.2:58043)
0924 00:19:22.659784 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:22.659827 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:22.660469 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:22.660746 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:22.660750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:22.661014 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:22.661215 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:22.661220 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.661977 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.661978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:22.662918 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:22.662922 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.663343 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.663344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:22.663520 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.664163 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:22.664209 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:22.666016 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:22.667995 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:22.668002 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:22.668004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:22.668320 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:22.668541 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:22.668542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:22.668542 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:22.668656 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:22.668920 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:22.668927 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:22.669070 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 00:19:22.669171 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 00:19:22.669329 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":119,"threads_started":1}
I20260924 00:19:22.670073 2466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45675
I20260924 00:19:22.670177 2466 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:22.670354 2466 heartbeater.cc:511] Master 127.0.98.62:45675 requested a full tablet report, sending...
I20260924 00:19:22.670714 2123 ts_manager.cc:194] Registered new tserver with Master: f59aa39d2fa147cc8faf18842d35958e (127.0.98.2:32917)
I20260924 00:19:22.671136 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58043
I20260924 00:19:22.675124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 2335
I20260924 00:19:22.675491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:19:22.675735 392 external_mini_cluster.cc:1428] Setting key d941c42a1da62f3107af36b6e60b15a7
I20260924 00:19:22.677474 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.671165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48829 (local address 127.0.98.62:45675)
0924 00:19:22.671211 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:22.671213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:22.671371 (+ 158us) server_negotiation.cc:407] Connection header received
0924 00:19:22.671455 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:22.671456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:22.671466 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:22.671521 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:22.671837 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.672380 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.673094 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.673324 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.673416 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.674459 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:22.674460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:22.674461 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:22.674468 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:22.676142 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:22.676599 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:22.676600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:22.676600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:22.676658 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:22.676913 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:22.676913 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:22.676914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:22.677047 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:22.677097 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:22.677282 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 00:19:22.677337 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 00:19:22.677596 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.671111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45675 (local address 127.0.98.2:48829)
0924 00:19:22.671302 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:22.671311 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:22.671396 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:22.671527 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:22.671527 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:22.671703 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:22.671753 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:22.671754 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.672468 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.672469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:22.672995 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:22.672997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.673418 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.673419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:22.673544 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.674057 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:22.674064 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:22.674350 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:22.676262 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:22.676263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:22.676264 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:22.676503 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:22.676738 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:22.676739 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:22.676739 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:22.676833 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:22.677115 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:22.677116 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:22.677175 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 00:19:22.677264 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 00:19:22.677413 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:19:22.704523 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45675
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:22.789314 2475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:22.789577 2475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:22.789623 2475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:22.790834 2475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:22.790908 2475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:22.790952 2475 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 00:19:22.790984 2475 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:19:22.791023 2475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:19:22.792613 2475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:22.792922 2475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:22.793171 2475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:22.798498 2480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:22.798557 2481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:22.798539 2483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:22.798720 2475 server_base.cc:1034] running on GCE node
I20260924 00:19:22.799024 2475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:22.799302 2475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:22.800460 2475 hybrid_clock.cc:648] HybridClock initialized: now 1790209162800429 us; error 37 us; skew 500 ppm
Sep 24 00:19:22 dist-test-slave-777x krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:22.803188 2475 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:19:22.803721 2475 webserver.cc:460] Webserver started at http://127.0.98.3:38477/ using document root <none> and password file <none>
I20260924 00:19:22.803982 2475 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:22.804057 2475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:22.804275 2475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:22.805472 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "a4cf5c98df9a431e96a03b29534699cf"
format_stamp: "Formatted at 2026-09-24 00:19:22 on dist-test-slave-777x"
server_key: "88dced083fbda10a3eb360ddb527f52c"
server_key_iv: "14802fe471b4286d93b604ad7b76dfcd"
server_key_version: "encryptionkey@0"
I20260924 00:19:22.805914 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a4cf5c98df9a431e96a03b29534699cf"
format_stamp: "Formatted at 2026-09-24 00:19:22 on dist-test-slave-777x"
server_key: "88dced083fbda10a3eb360ddb527f52c"
server_key_iv: "14802fe471b4286d93b604ad7b76dfcd"
server_key_version: "encryptionkey@0"
I20260924 00:19:22.807355 2475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:19:22.808128 2490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:22.808367 2475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:22.808481 2475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "a4cf5c98df9a431e96a03b29534699cf"
format_stamp: "Formatted at 2026-09-24 00:19:22 on dist-test-slave-777x"
server_key: "88dced083fbda10a3eb360ddb527f52c"
server_key_iv: "14802fe471b4286d93b604ad7b76dfcd"
server_key_version: "encryptionkey@0"
I20260924 00:19:22.808559 2475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:22.831050 2475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:22.833520 2475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:22.833705 2475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:22.834216 2475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:22.834276 2475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:22.834327 2475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:22.834380 2475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 00:19:22 dist-test-slave-777x krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:22.841516 2475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39273
I20260924 00:19:22.841565 2605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39273 every 8 connection(s)
I20260924 00:19:22.841924 2475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:19:22.844671 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.834846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36075 (local address 127.0.98.62:45675)
0924 00:19:22.834911 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:22.834913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:22.835155 (+ 242us) server_negotiation.cc:407] Connection header received
0924 00:19:22.835662 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:22.835663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:22.835678 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:22.835761 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:22.836746 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.837678 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.838812 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.839168 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.839253 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.841832 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:22.841833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:22.841833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:22.841841 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:22.843182 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:22.843580 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:22.843581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:22.843581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:22.843648 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:22.844007 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:22.844007 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:22.844008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:22.844130 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:22.844251 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:22.844533 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 00:19:22.844605 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 00:19:22.844898 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.834843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45675 (local address 127.0.98.3:36075)
0924 00:19:22.835041 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:22.835076 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:22.835553 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:22.835819 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:22.835823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:22.836302 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:22.836632 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:22.836640 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.837810 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.837812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:22.838741 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:22.838744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.839272 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.839273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:22.839380 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.840041 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:22.840051 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:22.841740 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:22.843255 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:22.843260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:22.843262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:22.843492 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:22.843766 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:22.843767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:22.843767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:22.843875 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:22.844267 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:22.844280 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:22.844401 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 00:19:22.844504 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 00:19:22.844631 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":112,"threads_started":1}
I20260924 00:19:22.845578 2606 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45675
I20260924 00:19:22.845659 2606 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:22.845911 2606 heartbeater.cc:511] Master 127.0.98.62:45675 requested a full tablet report, sending...
I20260924 00:19:22.846333 2123 ts_manager.cc:194] Registered new tserver with Master: a4cf5c98df9a431e96a03b29534699cf (127.0.98.3:39273)
I20260924 00:19:22.846834 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36075
I20260924 00:19:22.850096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 2475
I20260924 00:19:22.850234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:19:22.850445 392 external_mini_cluster.cc:1428] Setting key a2f6c72215978b2014994af79f0ddf06
I20260924 00:19:22.851454 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:19:22.852809 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.846731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35895 (local address 127.0.98.62:45675)
0924 00:19:22.846787 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:22.846789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:22.846841 (+ 52us) server_negotiation.cc:407] Connection header received
0924 00:19:22.846947 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:22.846948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:22.846959 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:22.847031 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:22.847341 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.847878 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.848653 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.848919 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.849003 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.849936 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:22.849938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:22.849938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:22.849945 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:22.851647 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:22.851986 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:22.851987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:22.851987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:22.852042 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:22.852262 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:22.852263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:22.852263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:22.852375 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:22.852488 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:22.852644 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 00:19:22.852718 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 00:19:22.852824 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.846637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45675 (local address 127.0.98.3:35895)
0924 00:19:22.846785 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:22.846796 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:22.846871 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:22.847046 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:22.847047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:22.847215 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:22.847263 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:22.847265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.847972 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.847973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:22.848559 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:22.848560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.849005 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.849006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:22.849082 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.849550 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:22.849559 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:22.849826 (+ 267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:22.851737 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:22.851737 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:22.851738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:22.851907 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:22.852116 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:22.852117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:22.852117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:22.852202 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:22.852482 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:22.852483 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:22.852529 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 00:19:22.852616 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 00:19:22.852719 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":100,"threads_started":1}
Sep 24 00:19:22 dist-test-slave-777x krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 00:19:22 dist-test-slave-777x krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:22.874758 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.867474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58050 (local address 127.0.98.62:45675)
0924 00:19:22.867546 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:22.867548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:22.867557 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:19:22.867582 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:22.867582 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:22.867591 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:22.867642 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:22.868453 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.868965 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.869705 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.869897 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.869983 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.872257 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:22.872258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:22.872259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:22.872269 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:22.873544 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:22.873947 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:22.873949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:22.873949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:22.874009 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:22.874216 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:22.874217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:22.874217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:22.874336 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:22.874444 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:22.874569 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 00:19:22.874651 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 00:19:22.876425 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:19:22.878185 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:19:22.884912 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.880582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39273 (local address 127.0.0.1:34760)
0924 00:19:22.880814 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:22.880858 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:22.880926 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:22.881227 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:22.881230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:22.881232 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:22.881303 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:22.881305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.882410 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.882421 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:22.883608 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:22.883610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.884577 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.884578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:22.884663 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.884678 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 00:19:22.884759 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 00:19:22.884815 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":61,"threads_started":1}
I20260924 00:19:22.884920 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.880766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34760 (local address 127.0.98.3:39273)
0924 00:19:22.880978 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:22.880983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:22.880994 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:19:22.881036 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:22.881039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:22.881057 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:22.881149 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:22.881394 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.882306 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.883699 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.884493 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.884569 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.884627 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:22.884785 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 00:19:22.884822 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":88,"threads_started":1}
I20260924 00:19:22.885490 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.880652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40309 (local address 127.0.0.1:52368)
0924 00:19:22.881027 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:22.881036 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:22.881072 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:22.881607 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:22.881607 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:22.881610 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:22.881660 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:22.881663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.882965 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.882965 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:22.884146 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:22.884149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.885200 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.885201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:22.885277 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.885280 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:19:22.885361 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 00:19:22.885410 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":24,"thread_start_us":82,"threads_started":1}
I20260924 00:19:22.885512 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.880757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52368 (local address 127.0.98.1:40309)
0924 00:19:22.881373 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:22.881377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:22.881398 (+ 21us) server_negotiation.cc:407] Connection header received
0924 00:19:22.881438 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:22.881440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:22.881456 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:22.881533 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:22.881719 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.882856 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.884288 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.885041 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.885118 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.885170 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:22.885382 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 00:19:22.885421 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":95,"threads_started":1}
I20260924 00:19:22.885808 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.880780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32917 (local address 127.0.0.1:33626)
0924 00:19:22.881181 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:22.881191 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:22.881251 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:22.881624 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:22.881624 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:22.881627 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:22.881685 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:22.881686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.882950 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.882950 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:22.884045 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:22.884063 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.884982 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.884983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:22.885490 (+ 507us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.885495 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:19:22.885551 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 00:19:22.885739 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":72,"thread_start_us":64,"threads_started":1}
I20260924 00:19:22.885927 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.880995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33626 (local address 127.0.98.2:32917)
0924 00:19:22.881163 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:22.881168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:22.881458 (+ 290us) server_negotiation.cc:407] Connection header received
0924 00:19:22.881521 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:22.881524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:22.881539 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:22.881625 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:22.881761 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.882856 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.884130 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.884822 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.884894 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.884947 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:22.885778 (+ 831us) server_negotiation.cc:299] Negotiation successful
0924 00:19:22.885822 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":102,"threads_started":1}
I20260924 00:19:22.885862 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 7675270f13dc47508bedf2343d1f918b (DEFAULT_TABLE table=test-table [id=3ea5e375ac554fcc89351b51d320377b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:22.886263 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7675270f13dc47508bedf2343d1f918b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:22.886683 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 7675270f13dc47508bedf2343d1f918b (DEFAULT_TABLE table=test-table [id=3ea5e375ac554fcc89351b51d320377b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:22.886727 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 7675270f13dc47508bedf2343d1f918b (DEFAULT_TABLE table=test-table [id=3ea5e375ac554fcc89351b51d320377b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:22.887035 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7675270f13dc47508bedf2343d1f918b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:22.887055 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7675270f13dc47508bedf2343d1f918b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:22.889775 2629 tablet_bootstrap.cc:492] T 7675270f13dc47508bedf2343d1f918b P a4cf5c98df9a431e96a03b29534699cf: Bootstrap starting.
I20260924 00:19:22.890122 2630 tablet_bootstrap.cc:492] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa: Bootstrap starting.
I20260924 00:19:22.890784 2630 tablet_bootstrap.cc:654] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:22.890789 2629 tablet_bootstrap.cc:654] T 7675270f13dc47508bedf2343d1f918b P a4cf5c98df9a431e96a03b29534699cf: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:22.890863 2631 tablet_bootstrap.cc:492] T 7675270f13dc47508bedf2343d1f918b P f59aa39d2fa147cc8faf18842d35958e: Bootstrap starting.
I20260924 00:19:22.891119 2630 log.cc:824] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:22.891320 2629 log.cc:824] T 7675270f13dc47508bedf2343d1f918b P a4cf5c98df9a431e96a03b29534699cf: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:22.891475 2631 tablet_bootstrap.cc:654] T 7675270f13dc47508bedf2343d1f918b P f59aa39d2fa147cc8faf18842d35958e: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:22.891805 2631 log.cc:824] T 7675270f13dc47508bedf2343d1f918b P f59aa39d2fa147cc8faf18842d35958e: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:22.891897 2630 tablet_bootstrap.cc:492] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa: No bootstrap required, opened a new log
I20260924 00:19:22.891956 2630 ts_tablet_manager.cc:1397] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:19:22.892088 2629 tablet_bootstrap.cc:492] T 7675270f13dc47508bedf2343d1f918b P a4cf5c98df9a431e96a03b29534699cf: No bootstrap required, opened a new log
I20260924 00:19:22.892136 2629 ts_tablet_manager.cc:1397] T 7675270f13dc47508bedf2343d1f918b P a4cf5c98df9a431e96a03b29534699cf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:19:22.892501 2631 tablet_bootstrap.cc:492] T 7675270f13dc47508bedf2343d1f918b P f59aa39d2fa147cc8faf18842d35958e: No bootstrap required, opened a new log
I20260924 00:19:22.892586 2631 ts_tablet_manager.cc:1397] T 7675270f13dc47508bedf2343d1f918b P f59aa39d2fa147cc8faf18842d35958e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:19:22.894054 2631 raft_consensus.cc:348] T 7675270f13dc47508bedf2343d1f918b P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } } peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } }
I20260924 00:19:22.894253 2631 raft_consensus.cc:374] T 7675270f13dc47508bedf2343d1f918b P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:22.894145 2630 raft_consensus.cc:348] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } } peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } }
I20260924 00:19:22.894145 2629 raft_consensus.cc:348] T 7675270f13dc47508bedf2343d1f918b P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } } peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } }
I20260924 00:19:22.894309 2629 raft_consensus.cc:374] T 7675270f13dc47508bedf2343d1f918b P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:22.894309 2630 raft_consensus.cc:374] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:22.894306 2631 raft_consensus.cc:729] T 7675270f13dc47508bedf2343d1f918b P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f59aa39d2fa147cc8faf18842d35958e, State: Initialized, Role: FOLLOWER
I20260924 00:19:22.894345 2630 raft_consensus.cc:729] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b78a425f984265aa22d4a1917f23aa, State: Initialized, Role: FOLLOWER
I20260924 00:19:22.894345 2629 raft_consensus.cc:729] T 7675270f13dc47508bedf2343d1f918b P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cf5c98df9a431e96a03b29534699cf, State: Initialized, Role: FOLLOWER
I20260924 00:19:22.894472 2630 consensus_queue.cc:260] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } } peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } }
I20260924 00:19:22.894475 2629 consensus_queue.cc:260] T 7675270f13dc47508bedf2343d1f918b P a4cf5c98df9a431e96a03b29534699cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } } peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } }
I20260924 00:19:22.894505 2631 consensus_queue.cc:260] T 7675270f13dc47508bedf2343d1f918b P f59aa39d2fa147cc8faf18842d35958e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } } peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } }
I20260924 00:19:22.894781 2630 ts_tablet_manager.cc:1428] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:19:22.894788 2631 ts_tablet_manager.cc:1428] T 7675270f13dc47508bedf2343d1f918b P f59aa39d2fa147cc8faf18842d35958e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:19:22.894822 2466 heartbeater.cc:503] Master 127.0.98.62:45675 was elected leader, sending a full tablet report...
I20260924 00:19:22.894932 2326 heartbeater.cc:503] Master 127.0.98.62:45675 was elected leader, sending a full tablet report...
I20260924 00:19:22.895056 2606 heartbeater.cc:503] Master 127.0.98.62:45675 was elected leader, sending a full tablet report...
I20260924 00:19:22.895639 2629 ts_tablet_manager.cc:1428] T 7675270f13dc47508bedf2343d1f918b P a4cf5c98df9a431e96a03b29534699cf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260924 00:19:22.917789 2467 tablet.cc:2369] T 7675270f13dc47508bedf2343d1f918b P f59aa39d2fa147cc8faf18842d35958e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:19:22.974007 2636 raft_consensus.cc:482] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:19:22.974176 2636 raft_consensus.cc:504] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } } peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } }
I20260924 00:19:22.974593 2636 leader_election.cc:288] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4cf5c98df9a431e96a03b29534699cf (127.0.98.3:39273), f59aa39d2fa147cc8faf18842d35958e (127.0.98.2:32917)
I20260924 00:19:22.977932 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.974936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39273 (local address 127.0.98.1:57243)
0924 00:19:22.975005 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:22.975018 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:22.975064 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:22.975380 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:22.975381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:22.975383 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:22.975468 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:22.975470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.976267 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.976268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:22.977066 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:22.977068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.977663 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.977664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:22.977741 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.977744 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:19:22.977803 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 00:19:22.977834 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":3}
I20260924 00:19:22.977944 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.975071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57243 (local address 127.0.98.3:39273)
0924 00:19:22.975126 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:22.975128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:22.975138 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:19:22.975170 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:22.975171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:22.975184 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:22.975336 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:22.975548 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.976122 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.977164 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.977575 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.977637 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.977673 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:22.977825 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 00:19:22.977872 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 00:19:22.978008 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.975190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35707 (local address 127.0.98.2:32917)
0924 00:19:22.975248 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:22.975250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:22.975371 (+ 121us) server_negotiation.cc:407] Connection header received
0924 00:19:22.975447 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:22.975448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:22.975459 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:22.975521 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:22.975696 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.976154 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.977210 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.977592 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.977680 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.977689 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:22.977892 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 00:19:22.977934 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 00:19:22.978286 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:22.975079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32917 (local address 127.0.98.1:35707)
0924 00:19:22.975286 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:22.975298 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:22.975364 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:22.975540 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:22.975541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:22.975542 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:22.975616 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:22.975618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.976272 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.976273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:22.977071 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:22.977074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:22.977706 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:22.977709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:22.977802 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:22.977807 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:19:22.977873 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 00:19:22.977913 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":39,"thread_start_us":126,"threads_started":1}
I20260924 00:19:22.978323 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7675270f13dc47508bedf2343d1f918b" candidate_uuid: "86b78a425f984265aa22d4a1917f23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cf5c98df9a431e96a03b29534699cf" is_pre_election: true
I20260924 00:19:22.978503 2560 raft_consensus.cc:2454] T 7675270f13dc47508bedf2343d1f918b P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86b78a425f984265aa22d4a1917f23aa in term 0.
I20260924 00:19:22.978658 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7675270f13dc47508bedf2343d1f918b" candidate_uuid: "86b78a425f984265aa22d4a1917f23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f59aa39d2fa147cc8faf18842d35958e" is_pre_election: true
I20260924 00:19:22.978798 2420 raft_consensus.cc:2454] T 7675270f13dc47508bedf2343d1f918b P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86b78a425f984265aa22d4a1917f23aa in term 0.
I20260924 00:19:22.978775 2211 leader_election.cc:302] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86b78a425f984265aa22d4a1917f23aa, a4cf5c98df9a431e96a03b29534699cf; no voters:
I20260924 00:19:22.978924 2636 raft_consensus.cc:2781] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:19:22.978988 2636 raft_consensus.cc:482] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:19:22.979017 2636 raft_consensus.cc:3037] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:22.979741 2636 raft_consensus.cc:504] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } } peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } }
I20260924 00:19:22.979899 2636 leader_election.cc:288] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa [CANDIDATE]: Term 1 election: Requested vote from peers a4cf5c98df9a431e96a03b29534699cf (127.0.98.3:39273), f59aa39d2fa147cc8faf18842d35958e (127.0.98.2:32917)
I20260924 00:19:22.980087 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7675270f13dc47508bedf2343d1f918b" candidate_uuid: "86b78a425f984265aa22d4a1917f23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cf5c98df9a431e96a03b29534699cf"
I20260924 00:19:22.980087 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7675270f13dc47508bedf2343d1f918b" candidate_uuid: "86b78a425f984265aa22d4a1917f23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f59aa39d2fa147cc8faf18842d35958e"
I20260924 00:19:22.980202 2420 raft_consensus.cc:3037] T 7675270f13dc47508bedf2343d1f918b P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:22.980202 2560 raft_consensus.cc:3037] T 7675270f13dc47508bedf2343d1f918b P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:22.981155 2560 raft_consensus.cc:2454] T 7675270f13dc47508bedf2343d1f918b P a4cf5c98df9a431e96a03b29534699cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86b78a425f984265aa22d4a1917f23aa in term 1.
I20260924 00:19:22.981155 2420 raft_consensus.cc:2454] T 7675270f13dc47508bedf2343d1f918b P f59aa39d2fa147cc8faf18842d35958e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86b78a425f984265aa22d4a1917f23aa in term 1.
I20260924 00:19:22.981348 2211 leader_election.cc:302] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86b78a425f984265aa22d4a1917f23aa, f59aa39d2fa147cc8faf18842d35958e; no voters:
I20260924 00:19:22.981478 2636 raft_consensus.cc:2781] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:22.981603 2636 raft_consensus.cc:686] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa [term 1 LEADER]: Becoming Leader. State: Replica: 86b78a425f984265aa22d4a1917f23aa, State: Running, Role: LEADER
I20260924 00:19:22.981699 2636 consensus_queue.cc:237] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } } peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } }
I20260924 00:19:22.982507 2121 catalog_manager.cc:5495] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 86b78a425f984265aa22d4a1917f23aa (127.0.98.1). New cstate: current_term: 1 leader_uuid: "86b78a425f984265aa22d4a1917f23aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } health_report { overall_health: UNKNOWN } } }
W20260924 00:19:23.010111 2327 tablet.cc:2369] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 00:19:23 dist-test-slave-777x krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:23.024641 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:23.017077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45675 (local address 127.0.0.1:58066)
0924 00:19:23.017309 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:23.017320 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:23.017404 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:23.017573 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:23.017574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:23.017933 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:23.017981 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:23.017984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:23.018670 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:23.018671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:23.019252 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:23.019254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:23.019618 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:23.019619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:23.019663 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:23.020277 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:23.020281 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:23.021734 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:23.023500 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:23.023507 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:23.023509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:23.023714 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:23.023919 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:23.023920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:23.023920 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:23.023978 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:23.024277 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:23.024284 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:23.024333 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 00:19:23.024436 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 00:19:23.024539 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":120,"threads_started":1}
I20260924 00:19:23.024727 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:23.017158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58066 (local address 127.0.98.62:45675)
0924 00:19:23.017238 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:23.017240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:23.017408 (+ 168us) server_negotiation.cc:407] Connection header received
0924 00:19:23.017451 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:23.017451 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:23.017463 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:23.017525 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:23.018079 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:23.018577 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:23.019348 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:23.019541 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:23.019606 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:23.021868 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:23.021870 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:23.021870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:23.021876 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:23.023412 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:23.023789 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:23.023790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:23.023790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:23.023853 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:23.024035 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:23.024035 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:23.024036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:23.024137 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:23.024263 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:23.024497 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 00:19:23.024592 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":48}
I20260924 00:19:23.027093 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58066:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 00:19:23.027324 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:19:23.029063 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 86d42f9e68e54745bbb0abb59ff69bf8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fecca9eba73141bc9622d306c4819628]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:19:23.029065 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 86d42f9e68e54745bbb0abb59ff69bf8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fecca9eba73141bc9622d306c4819628]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:19:23.029078 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 86d42f9e68e54745bbb0abb59ff69bf8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fecca9eba73141bc9622d306c4819628]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:19:23.029249 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86d42f9e68e54745bbb0abb59ff69bf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:23.029250 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86d42f9e68e54745bbb0abb59ff69bf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:23.029250 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86d42f9e68e54745bbb0abb59ff69bf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:23.030563 2630 tablet_bootstrap.cc:492] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa: Bootstrap starting.
I20260924 00:19:23.030781 2631 tablet_bootstrap.cc:492] T 86d42f9e68e54745bbb0abb59ff69bf8 P f59aa39d2fa147cc8faf18842d35958e: Bootstrap starting.
I20260924 00:19:23.030840 2629 tablet_bootstrap.cc:492] T 86d42f9e68e54745bbb0abb59ff69bf8 P a4cf5c98df9a431e96a03b29534699cf: Bootstrap starting.
I20260924 00:19:23.030982 2630 tablet_bootstrap.cc:654] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:23.031386 2629 tablet_bootstrap.cc:654] T 86d42f9e68e54745bbb0abb59ff69bf8 P a4cf5c98df9a431e96a03b29534699cf: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:23.031528 2631 tablet_bootstrap.cc:654] T 86d42f9e68e54745bbb0abb59ff69bf8 P f59aa39d2fa147cc8faf18842d35958e: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:23.032212 2630 tablet_bootstrap.cc:492] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa: No bootstrap required, opened a new log
I20260924 00:19:23.032218 2629 tablet_bootstrap.cc:492] T 86d42f9e68e54745bbb0abb59ff69bf8 P a4cf5c98df9a431e96a03b29534699cf: No bootstrap required, opened a new log
I20260924 00:19:23.032266 2630 ts_tablet_manager.cc:1397] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:19:23.032259 2629 ts_tablet_manager.cc:1397] T 86d42f9e68e54745bbb0abb59ff69bf8 P a4cf5c98df9a431e96a03b29534699cf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:23.032423 2630 raft_consensus.cc:348] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:23.032420 2629 raft_consensus.cc:348] T 86d42f9e68e54745bbb0abb59ff69bf8 P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:23.032501 2629 raft_consensus.cc:374] T 86d42f9e68e54745bbb0abb59ff69bf8 P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:23.032503 2630 raft_consensus.cc:374] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:23.032516 2629 raft_consensus.cc:729] T 86d42f9e68e54745bbb0abb59ff69bf8 P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cf5c98df9a431e96a03b29534699cf, State: Initialized, Role: FOLLOWER
I20260924 00:19:23.032528 2630 raft_consensus.cc:729] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b78a425f984265aa22d4a1917f23aa, State: Initialized, Role: FOLLOWER
I20260924 00:19:23.032584 2629 consensus_queue.cc:260] T 86d42f9e68e54745bbb0abb59ff69bf8 P a4cf5c98df9a431e96a03b29534699cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:23.032589 2630 consensus_queue.cc:260] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:23.032670 2629 ts_tablet_manager.cc:1428] T 86d42f9e68e54745bbb0abb59ff69bf8 P a4cf5c98df9a431e96a03b29534699cf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:23.032676 2630 ts_tablet_manager.cc:1428] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:23.032825 2635 tablet_replica.cc:439] T 86d42f9e68e54745bbb0abb59ff69bf8 P a4cf5c98df9a431e96a03b29534699cf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } } }
I20260924 00:19:23.032928 2635 tablet_replica.cc:442] T 86d42f9e68e54745bbb0abb59ff69bf8 P a4cf5c98df9a431e96a03b29534699cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:19:23.032954 2631 tablet_bootstrap.cc:492] T 86d42f9e68e54745bbb0abb59ff69bf8 P f59aa39d2fa147cc8faf18842d35958e: No bootstrap required, opened a new log
I20260924 00:19:23.033005 2631 ts_tablet_manager.cc:1397] T 86d42f9e68e54745bbb0abb59ff69bf8 P f59aa39d2fa147cc8faf18842d35958e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:19:23.033056 2636 tablet_replica.cc:439] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } } }
I20260924 00:19:23.033149 2636 tablet_replica.cc:442] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:19:23.033172 2631 raft_consensus.cc:348] T 86d42f9e68e54745bbb0abb59ff69bf8 P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:23.033248 2631 raft_consensus.cc:374] T 86d42f9e68e54745bbb0abb59ff69bf8 P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:23.033274 2631 raft_consensus.cc:729] T 86d42f9e68e54745bbb0abb59ff69bf8 P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f59aa39d2fa147cc8faf18842d35958e, State: Initialized, Role: FOLLOWER
I20260924 00:19:23.033331 2631 consensus_queue.cc:260] T 86d42f9e68e54745bbb0abb59ff69bf8 P f59aa39d2fa147cc8faf18842d35958e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:23.033418 2631 ts_tablet_manager.cc:1428] T 86d42f9e68e54745bbb0abb59ff69bf8 P f59aa39d2fa147cc8faf18842d35958e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 00:19:23.033496 2637 tablet_replica.cc:439] T 86d42f9e68e54745bbb0abb59ff69bf8 P f59aa39d2fa147cc8faf18842d35958e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } } }
I20260924 00:19:23.033603 2637 tablet_replica.cc:442] T 86d42f9e68e54745bbb0abb59ff69bf8 P f59aa39d2fa147cc8faf18842d35958e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:19:23.043012 2636 raft_consensus.cc:482] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:19:23.043113 2636 raft_consensus.cc:504] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:23.043293 2636 leader_election.cc:288] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f59aa39d2fa147cc8faf18842d35958e (127.0.98.2:32917), a4cf5c98df9a431e96a03b29534699cf (127.0.98.3:39273)
I20260924 00:19:23.043476 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d42f9e68e54745bbb0abb59ff69bf8" candidate_uuid: "86b78a425f984265aa22d4a1917f23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f59aa39d2fa147cc8faf18842d35958e" is_pre_election: true
I20260924 00:19:23.043588 2420 raft_consensus.cc:2454] T 86d42f9e68e54745bbb0abb59ff69bf8 P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86b78a425f984265aa22d4a1917f23aa in term 0.
I20260924 00:19:23.043527 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d42f9e68e54745bbb0abb59ff69bf8" candidate_uuid: "86b78a425f984265aa22d4a1917f23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cf5c98df9a431e96a03b29534699cf" is_pre_election: true
I20260924 00:19:23.043639 2560 raft_consensus.cc:2454] T 86d42f9e68e54745bbb0abb59ff69bf8 P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86b78a425f984265aa22d4a1917f23aa in term 0.
I20260924 00:19:23.043736 2211 leader_election.cc:302] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86b78a425f984265aa22d4a1917f23aa, f59aa39d2fa147cc8faf18842d35958e; no voters:
I20260924 00:19:23.043915 2636 raft_consensus.cc:2781] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:19:23.043985 2636 raft_consensus.cc:482] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:19:23.044018 2636 raft_consensus.cc:3037] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:23.044804 2636 raft_consensus.cc:504] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:23.044945 2636 leader_election.cc:288] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa [CANDIDATE]: Term 1 election: Requested vote from peers f59aa39d2fa147cc8faf18842d35958e (127.0.98.2:32917), a4cf5c98df9a431e96a03b29534699cf (127.0.98.3:39273)
I20260924 00:19:23.045156 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d42f9e68e54745bbb0abb59ff69bf8" candidate_uuid: "86b78a425f984265aa22d4a1917f23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cf5c98df9a431e96a03b29534699cf"
I20260924 00:19:23.045166 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86d42f9e68e54745bbb0abb59ff69bf8" candidate_uuid: "86b78a425f984265aa22d4a1917f23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f59aa39d2fa147cc8faf18842d35958e"
I20260924 00:19:23.045248 2560 raft_consensus.cc:3037] T 86d42f9e68e54745bbb0abb59ff69bf8 P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:23.045248 2420 raft_consensus.cc:3037] T 86d42f9e68e54745bbb0abb59ff69bf8 P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:23.046099 2560 raft_consensus.cc:2454] T 86d42f9e68e54745bbb0abb59ff69bf8 P a4cf5c98df9a431e96a03b29534699cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86b78a425f984265aa22d4a1917f23aa in term 1.
I20260924 00:19:23.046139 2420 raft_consensus.cc:2454] T 86d42f9e68e54745bbb0abb59ff69bf8 P f59aa39d2fa147cc8faf18842d35958e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86b78a425f984265aa22d4a1917f23aa in term 1.
I20260924 00:19:23.046307 2211 leader_election.cc:302] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86b78a425f984265aa22d4a1917f23aa, a4cf5c98df9a431e96a03b29534699cf; no voters:
I20260924 00:19:23.046466 2636 raft_consensus.cc:2781] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:23.046531 2636 raft_consensus.cc:686] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa [term 1 LEADER]: Becoming Leader. State: Replica: 86b78a425f984265aa22d4a1917f23aa, State: Running, Role: LEADER
I20260924 00:19:23.046609 2636 consensus_queue.cc:237] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:23.047030 2639 tablet_replica.cc:439] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa: TxnStatusTablet state changed. Reason: New leader 86b78a425f984265aa22d4a1917f23aa. Latest consensus state: current_term: 1 leader_uuid: "86b78a425f984265aa22d4a1917f23aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } } }
I20260924 00:19:23.047122 2639 tablet_replica.cc:442] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:19:23.047372 2651 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 00:19:23.047390 2121 catalog_manager.cc:5495] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 86b78a425f984265aa22d4a1917f23aa (127.0.98.1). New cstate: current_term: 1 leader_uuid: "86b78a425f984265aa22d4a1917f23aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } health_report { overall_health: UNKNOWN } } }
I20260924 00:19:23.047541 2651 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 00:19:23.057543 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:23.054795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40309 (local address 127.0.0.1:52370)
0924 00:19:23.054893 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:23.054907 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:23.054961 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:23.055196 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:23.055197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:23.055198 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:19:23.055251 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:23.055253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:23.055974 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:23.055976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:23.056641 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:23.056643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:23.057031 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:23.057032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:23.057101 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:23.057114 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:19:23.057371 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:19:23.057379 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 00:19:23.057429 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 00:19:23.057475 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 00:19:23.057541 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:23.054982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52370 (local address 127.0.98.1:40309)
0924 00:19:23.055052 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:23.055054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:23.055062 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:19:23.055099 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:23.055100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:23.055118 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:23.055192 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:23.055373 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:23.055860 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:23.056743 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:23.056943 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:23.057029 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:23.057205 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:23.057304 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:23.057397 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:23.057436 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 00:19:23.057459 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 00:19:23.059577 2420 raft_consensus.cc:1261] T 86d42f9e68e54745bbb0abb59ff69bf8 P f59aa39d2fa147cc8faf18842d35958e [term 1 FOLLOWER]: Refusing update from remote peer 86b78a425f984265aa22d4a1917f23aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:23.059625 2560 raft_consensus.cc:1261] T 86d42f9e68e54745bbb0abb59ff69bf8 P a4cf5c98df9a431e96a03b29534699cf [term 1 FOLLOWER]: Refusing update from remote peer 86b78a425f984265aa22d4a1917f23aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:23.059934 2639 consensus_queue.cc:1035] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:23.060058 2636 consensus_queue.cc:1035] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:23.061367 2635 tablet_replica.cc:439] T 86d42f9e68e54745bbb0abb59ff69bf8 P a4cf5c98df9a431e96a03b29534699cf: TxnStatusTablet state changed. Reason: New leader 86b78a425f984265aa22d4a1917f23aa. Latest consensus state: current_term: 1 leader_uuid: "86b78a425f984265aa22d4a1917f23aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } } }
I20260924 00:19:23.061367 2637 tablet_replica.cc:439] T 86d42f9e68e54745bbb0abb59ff69bf8 P f59aa39d2fa147cc8faf18842d35958e: TxnStatusTablet state changed. Reason: New leader 86b78a425f984265aa22d4a1917f23aa. Latest consensus state: current_term: 1 leader_uuid: "86b78a425f984265aa22d4a1917f23aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } } }
I20260924 00:19:23.061481 2635 tablet_replica.cc:442] T 86d42f9e68e54745bbb0abb59ff69bf8 P a4cf5c98df9a431e96a03b29534699cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:19:23.061480 2637 tablet_replica.cc:442] T 86d42f9e68e54745bbb0abb59ff69bf8 P f59aa39d2fa147cc8faf18842d35958e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:19:23.062528 2636 tablet_replica.cc:439] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "86b78a425f984265aa22d4a1917f23aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } } }
I20260924 00:19:23.062628 2636 tablet_replica.cc:442] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:19:23.062762 2654 mvcc.cc:205] Tried to move back new op lower bound from 7332696731889500160 to 7332696731839373312. Current Snapshot: MvccSnapshot[applied={T|T < 7332696731889500160}]
I20260924 00:19:23.062533 2639 tablet_replica.cc:439] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "86b78a425f984265aa22d4a1917f23aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } } }
I20260924 00:19:23.063102 2639 tablet_replica.cc:442] T 86d42f9e68e54745bbb0abb59ff69bf8 P 86b78a425f984265aa22d4a1917f23aa: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:19:23.063262 2637 tablet_replica.cc:439] T 86d42f9e68e54745bbb0abb59ff69bf8 P f59aa39d2fa147cc8faf18842d35958e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "86b78a425f984265aa22d4a1917f23aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } } }
I20260924 00:19:23.063362 2637 tablet_replica.cc:442] T 86d42f9e68e54745bbb0abb59ff69bf8 P f59aa39d2fa147cc8faf18842d35958e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:19:23.063911 2635 tablet_replica.cc:439] T 86d42f9e68e54745bbb0abb59ff69bf8 P a4cf5c98df9a431e96a03b29534699cf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "86b78a425f984265aa22d4a1917f23aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } } }
I20260924 00:19:23.063987 2635 tablet_replica.cc:442] T 86d42f9e68e54745bbb0abb59ff69bf8 P a4cf5c98df9a431e96a03b29534699cf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:19:23.064584 2655 mvcc.cc:205] Tried to move back new op lower bound from 7332696731889500160 to 7332696731839373312. Current Snapshot: MvccSnapshot[applied={T|T < 7332696731889500160 or (T in {7332696731889500160})}]
I20260924 00:19:23.064703 2656 mvcc.cc:205] Tried to move back new op lower bound from 7332696731889500160 to 7332696731839373312. Current Snapshot: MvccSnapshot[applied={T|T < 7332696731889500160}]
I20260924 00:19:23.067030 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:23.064538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52376 (local address 127.0.98.1:40309)
0924 00:19:23.064608 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:23.064610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:23.064618 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:19:23.064637 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:23.064638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:23.064647 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:23.064709 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:23.064962 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:23.065491 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:23.066320 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:23.066529 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:23.066599 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:23.066766 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:23.066797 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:23.066875 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:23.066938 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 00:19:23.066957 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 00:19:23.073010 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:23.069552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40309 (local address 127.0.98.1:57041)
0924 00:19:23.069713 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:23.069724 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:23.069779 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:23.069920 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:23.069921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:23.069932 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:19:23.070002 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:23.070003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:23.070635 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:23.070636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:23.071996 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:23.071999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:23.072456 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:23.072457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:23.072536 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:23.072550 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:19:23.072782 (+ 232us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:19:23.072784 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:19:23.072832 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 00:19:23.072878 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260924 00:19:23.073022 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:23.069608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57041 (local address 127.0.98.1:40309)
0924 00:19:23.069642 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:23.069643 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:23.069772 (+ 129us) server_negotiation.cc:407] Connection header received
0924 00:19:23.069835 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:23.069836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:23.069845 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:23.069901 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:23.070061 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:23.070559 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:23.072093 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:23.072380 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:23.072442 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:23.072671 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:23.072714 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:23.072763 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:23.072884 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 00:19:23.072899 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 00:19:23.075435 2420 raft_consensus.cc:1261] T 7675270f13dc47508bedf2343d1f918b P f59aa39d2fa147cc8faf18842d35958e [term 1 FOLLOWER]: Refusing update from remote peer 86b78a425f984265aa22d4a1917f23aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:23.075430 2560 raft_consensus.cc:1261] T 7675270f13dc47508bedf2343d1f918b P a4cf5c98df9a431e96a03b29534699cf [term 1 FOLLOWER]: Refusing update from remote peer 86b78a425f984265aa22d4a1917f23aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:23.075702 2659 consensus_queue.cc:1035] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:23.075827 2639 consensus_queue.cc:1035] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:23.087626 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:23.087739 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:23.088527 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6: Sending DeleteTablet for 3 replicas of tablet 7675270f13dc47508bedf2343d1f918b
I20260924 00:19:23.088840 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 7675270f13dc47508bedf2343d1f918b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52368
I20260924 00:19:23.088863 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 7675270f13dc47508bedf2343d1f918b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34760
I20260924 00:19:23.088903 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 7675270f13dc47508bedf2343d1f918b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:23 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33626
I20260924 00:19:23.089097 2671 tablet_replica.cc:330] T 7675270f13dc47508bedf2343d1f918b P a4cf5c98df9a431e96a03b29534699cf: stopping tablet replica
I20260924 00:19:23.089216 2670 tablet_replica.cc:330] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa: stopping tablet replica
I20260924 00:19:23.089236 2671 raft_consensus.cc:2229] T 7675270f13dc47508bedf2343d1f918b P a4cf5c98df9a431e96a03b29534699cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:23.089367 2672 tablet_replica.cc:330] T 7675270f13dc47508bedf2343d1f918b P f59aa39d2fa147cc8faf18842d35958e: stopping tablet replica
I20260924 00:19:23.089370 2670 raft_consensus.cc:2229] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:19:23.089326 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 00:19:23.089466 2670 raft_consensus.cc:2258] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 00:19:23.089468 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:19:23.089696 2672 raft_consensus.cc:2229] T 7675270f13dc47508bedf2343d1f918b P f59aa39d2fa147cc8faf18842d35958e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:23.089824 2672 raft_consensus.cc:2258] T 7675270f13dc47508bedf2343d1f918b P f59aa39d2fa147cc8faf18842d35958e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:23.089900 2670 ts_tablet_manager.cc:1905] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:23.090171 2671 raft_consensus.cc:2258] T 7675270f13dc47508bedf2343d1f918b P a4cf5c98df9a431e96a03b29534699cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:23.090353 2672 ts_tablet_manager.cc:1905] T 7675270f13dc47508bedf2343d1f918b P f59aa39d2fa147cc8faf18842d35958e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:23.090890 2671 ts_tablet_manager.cc:1905] T 7675270f13dc47508bedf2343d1f918b P a4cf5c98df9a431e96a03b29534699cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:23.091005 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 6ed51afc59df441a8b31d95456d61108 (DEFAULT_TABLE table=test-table [id=d12702723e90476eaa39054f91b2df86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:23.091107 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ed51afc59df441a8b31d95456d61108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:23.091171 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 6ed51afc59df441a8b31d95456d61108 (DEFAULT_TABLE table=test-table [id=d12702723e90476eaa39054f91b2df86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:23.091328 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ed51afc59df441a8b31d95456d61108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:23.092370 2630 tablet_bootstrap.cc:492] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa: Bootstrap starting.
I20260924 00:19:23.091070 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 6ed51afc59df441a8b31d95456d61108 (DEFAULT_TABLE table=test-table [id=d12702723e90476eaa39054f91b2df86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:23.092587 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ed51afc59df441a8b31d95456d61108. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:23.092665 2629 tablet_bootstrap.cc:492] T 6ed51afc59df441a8b31d95456d61108 P a4cf5c98df9a431e96a03b29534699cf: Bootstrap starting.
I20260924 00:19:23.092799 2630 tablet_bootstrap.cc:654] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:23.093067 2629 tablet_bootstrap.cc:654] T 6ed51afc59df441a8b31d95456d61108 P a4cf5c98df9a431e96a03b29534699cf: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:23.093657 2629 tablet_bootstrap.cc:492] T 6ed51afc59df441a8b31d95456d61108 P a4cf5c98df9a431e96a03b29534699cf: No bootstrap required, opened a new log
I20260924 00:19:23.093707 2629 ts_tablet_manager.cc:1397] T 6ed51afc59df441a8b31d95456d61108 P a4cf5c98df9a431e96a03b29534699cf: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:23.093793 2671 ts_tablet_manager.cc:1918] T 7675270f13dc47508bedf2343d1f918b P a4cf5c98df9a431e96a03b29534699cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:19:23.093844 2671 log.cc:1196] T 7675270f13dc47508bedf2343d1f918b P a4cf5c98df9a431e96a03b29534699cf: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/7675270f13dc47508bedf2343d1f918b
I20260924 00:19:23.093863 2629 raft_consensus.cc:348] T 6ed51afc59df441a8b31d95456d61108 P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:23.093940 2629 raft_consensus.cc:374] T 6ed51afc59df441a8b31d95456d61108 P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:23.093943 2631 tablet_bootstrap.cc:492] T 6ed51afc59df441a8b31d95456d61108 P f59aa39d2fa147cc8faf18842d35958e: Bootstrap starting.
I20260924 00:19:23.093966 2629 raft_consensus.cc:729] T 6ed51afc59df441a8b31d95456d61108 P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cf5c98df9a431e96a03b29534699cf, State: Initialized, Role: FOLLOWER
I20260924 00:19:23.094013 2629 consensus_queue.cc:260] T 6ed51afc59df441a8b31d95456d61108 P a4cf5c98df9a431e96a03b29534699cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:23.094094 2671 ts_tablet_manager.cc:1939] T 7675270f13dc47508bedf2343d1f918b P a4cf5c98df9a431e96a03b29534699cf: Deleting consensus metadata
I20260924 00:19:23.094117 2629 ts_tablet_manager.cc:1428] T 6ed51afc59df441a8b31d95456d61108 P a4cf5c98df9a431e96a03b29534699cf: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:23.094298 2631 tablet_bootstrap.cc:654] T 6ed51afc59df441a8b31d95456d61108 P f59aa39d2fa147cc8faf18842d35958e: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:23.094504 2630 tablet_bootstrap.cc:492] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa: No bootstrap required, opened a new log
I20260924 00:19:23.094551 2630 ts_tablet_manager.cc:1397] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:19:23.094676 2630 raft_consensus.cc:348] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:23.094748 2630 raft_consensus.cc:374] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:23.094769 2630 raft_consensus.cc:729] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b78a425f984265aa22d4a1917f23aa, State: Initialized, Role: FOLLOWER
I20260924 00:19:23.094811 2630 consensus_queue.cc:260] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:23.094910 2630 ts_tablet_manager.cc:1428] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:23.096068 2672 ts_tablet_manager.cc:1918] T 7675270f13dc47508bedf2343d1f918b P f59aa39d2fa147cc8faf18842d35958e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:19:23.096145 2672 log.cc:1196] T 7675270f13dc47508bedf2343d1f918b P f59aa39d2fa147cc8faf18842d35958e: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-1/wal/wals/7675270f13dc47508bedf2343d1f918b
I20260924 00:19:23.096415 2672 ts_tablet_manager.cc:1939] T 7675270f13dc47508bedf2343d1f918b P f59aa39d2fa147cc8faf18842d35958e: Deleting consensus metadata
I20260924 00:19:23.096962 2107 catalog_manager.cc:4841] TS f59aa39d2fa147cc8faf18842d35958e (127.0.98.2:32917): tablet 7675270f13dc47508bedf2343d1f918b (table test-table [id=3ea5e375ac554fcc89351b51d320377b]) successfully deleted
I20260924 00:19:23.097882 2670 ts_tablet_manager.cc:1918] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:19:23.097956 2670 log.cc:1196] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-0/wal/wals/7675270f13dc47508bedf2343d1f918b
I20260924 00:19:23.098193 2670 ts_tablet_manager.cc:1939] T 7675270f13dc47508bedf2343d1f918b P 86b78a425f984265aa22d4a1917f23aa: Deleting consensus metadata
I20260924 00:19:23.098641 2109 catalog_manager.cc:4841] TS 86b78a425f984265aa22d4a1917f23aa (127.0.98.1:40309): tablet 7675270f13dc47508bedf2343d1f918b (table test-table [id=3ea5e375ac554fcc89351b51d320377b]) successfully deleted
I20260924 00:19:23.098742 2631 tablet_bootstrap.cc:492] T 6ed51afc59df441a8b31d95456d61108 P f59aa39d2fa147cc8faf18842d35958e: No bootstrap required, opened a new log
I20260924 00:19:23.098796 2631 ts_tablet_manager.cc:1397] T 6ed51afc59df441a8b31d95456d61108 P f59aa39d2fa147cc8faf18842d35958e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 00:19:23.098975 2107 catalog_manager.cc:4841] TS a4cf5c98df9a431e96a03b29534699cf (127.0.98.3:39273): tablet 7675270f13dc47508bedf2343d1f918b (table test-table [id=3ea5e375ac554fcc89351b51d320377b]) successfully deleted
I20260924 00:19:23.098975 2631 raft_consensus.cc:348] T 6ed51afc59df441a8b31d95456d61108 P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:23.099044 2631 raft_consensus.cc:374] T 6ed51afc59df441a8b31d95456d61108 P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:23.099071 2631 raft_consensus.cc:729] T 6ed51afc59df441a8b31d95456d61108 P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f59aa39d2fa147cc8faf18842d35958e, State: Initialized, Role: FOLLOWER
I20260924 00:19:23.099153 2631 consensus_queue.cc:260] T 6ed51afc59df441a8b31d95456d61108 P f59aa39d2fa147cc8faf18842d35958e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:23.099326 2631 ts_tablet_manager.cc:1428] T 6ed51afc59df441a8b31d95456d61108 P f59aa39d2fa147cc8faf18842d35958e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:23.315268 2639 raft_consensus.cc:482] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:19:23.315384 2639 raft_consensus.cc:504] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:23.315542 2639 leader_election.cc:288] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f59aa39d2fa147cc8faf18842d35958e (127.0.98.2:32917), a4cf5c98df9a431e96a03b29534699cf (127.0.98.3:39273)
I20260924 00:19:23.315805 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed51afc59df441a8b31d95456d61108" candidate_uuid: "86b78a425f984265aa22d4a1917f23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f59aa39d2fa147cc8faf18842d35958e" is_pre_election: true
I20260924 00:19:23.315852 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed51afc59df441a8b31d95456d61108" candidate_uuid: "86b78a425f984265aa22d4a1917f23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cf5c98df9a431e96a03b29534699cf" is_pre_election: true
I20260924 00:19:23.315932 2420 raft_consensus.cc:2454] T 6ed51afc59df441a8b31d95456d61108 P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86b78a425f984265aa22d4a1917f23aa in term 0.
I20260924 00:19:23.315936 2559 raft_consensus.cc:2454] T 6ed51afc59df441a8b31d95456d61108 P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86b78a425f984265aa22d4a1917f23aa in term 0.
I20260924 00:19:23.316126 2211 leader_election.cc:302] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86b78a425f984265aa22d4a1917f23aa, f59aa39d2fa147cc8faf18842d35958e; no voters:
I20260924 00:19:23.316299 2639 raft_consensus.cc:2781] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:19:23.316362 2639 raft_consensus.cc:482] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:19:23.316388 2639 raft_consensus.cc:3037] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:23.317167 2639 raft_consensus.cc:504] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:23.317332 2639 leader_election.cc:288] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa [CANDIDATE]: Term 1 election: Requested vote from peers f59aa39d2fa147cc8faf18842d35958e (127.0.98.2:32917), a4cf5c98df9a431e96a03b29534699cf (127.0.98.3:39273)
I20260924 00:19:23.317576 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed51afc59df441a8b31d95456d61108" candidate_uuid: "86b78a425f984265aa22d4a1917f23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f59aa39d2fa147cc8faf18842d35958e"
I20260924 00:19:23.317606 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ed51afc59df441a8b31d95456d61108" candidate_uuid: "86b78a425f984265aa22d4a1917f23aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cf5c98df9a431e96a03b29534699cf"
I20260924 00:19:23.317675 2559 raft_consensus.cc:3037] T 6ed51afc59df441a8b31d95456d61108 P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:23.317675 2420 raft_consensus.cc:3037] T 6ed51afc59df441a8b31d95456d61108 P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:23.318557 2420 raft_consensus.cc:2454] T 6ed51afc59df441a8b31d95456d61108 P f59aa39d2fa147cc8faf18842d35958e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86b78a425f984265aa22d4a1917f23aa in term 1.
I20260924 00:19:23.318599 2559 raft_consensus.cc:2454] T 6ed51afc59df441a8b31d95456d61108 P a4cf5c98df9a431e96a03b29534699cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86b78a425f984265aa22d4a1917f23aa in term 1.
I20260924 00:19:23.318773 2211 leader_election.cc:302] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86b78a425f984265aa22d4a1917f23aa, f59aa39d2fa147cc8faf18842d35958e; no voters:
I20260924 00:19:23.318890 2639 raft_consensus.cc:2781] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:23.318962 2639 raft_consensus.cc:686] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa [term 1 LEADER]: Becoming Leader. State: Replica: 86b78a425f984265aa22d4a1917f23aa, State: Running, Role: LEADER
I20260924 00:19:23.319024 2639 consensus_queue.cc:237] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:23.319654 2122 catalog_manager.cc:5495] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 86b78a425f984265aa22d4a1917f23aa (127.0.98.1). New cstate: current_term: 1 leader_uuid: "86b78a425f984265aa22d4a1917f23aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } health_report { overall_health: UNKNOWN } } }
W20260924 00:19:23.343149 2607 tablet.cc:2369] T 6ed51afc59df441a8b31d95456d61108 P a4cf5c98df9a431e96a03b29534699cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:19:23.382459 2559 raft_consensus.cc:1261] T 6ed51afc59df441a8b31d95456d61108 P a4cf5c98df9a431e96a03b29534699cf [term 1 FOLLOWER]: Refusing update from remote peer 86b78a425f984265aa22d4a1917f23aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:23.382452 2420 raft_consensus.cc:1261] T 6ed51afc59df441a8b31d95456d61108 P f59aa39d2fa147cc8faf18842d35958e [term 1 FOLLOWER]: Refusing update from remote peer 86b78a425f984265aa22d4a1917f23aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:23.382723 2636 consensus_queue.cc:1035] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:23.382866 2659 consensus_queue.cc:1035] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 00:19:23.388240 2654 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 00:19:24.913497 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:24.913657 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:24.914654 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6: Sending DeleteTablet for 3 replicas of tablet 6ed51afc59df441a8b31d95456d61108
I20260924 00:19:24.914960 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ed51afc59df441a8b31d95456d61108 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33626
I20260924 00:19:24.915004 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ed51afc59df441a8b31d95456d61108 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52368
I20260924 00:19:24.915159 2698 tablet_replica.cc:330] T 6ed51afc59df441a8b31d95456d61108 P f59aa39d2fa147cc8faf18842d35958e: stopping tablet replica
I20260924 00:19:24.915153 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ed51afc59df441a8b31d95456d61108 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34760
I20260924 00:19:24.915210 2698 raft_consensus.cc:2229] T 6ed51afc59df441a8b31d95456d61108 P f59aa39d2fa147cc8faf18842d35958e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:24.915223 2699 tablet_replica.cc:330] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa: stopping tablet replica
I20260924 00:19:24.915280 2699 raft_consensus.cc:2229] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:19:24.915292 2698 raft_consensus.cc:2258] T 6ed51afc59df441a8b31d95456d61108 P f59aa39d2fa147cc8faf18842d35958e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:24.915374 2699 raft_consensus.cc:2258] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:24.915714 2699 ts_tablet_manager.cc:1905] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:24.917520 2698 ts_tablet_manager.cc:1905] T 6ed51afc59df441a8b31d95456d61108 P f59aa39d2fa147cc8faf18842d35958e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:24.917697 2699 ts_tablet_manager.cc:1918] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 00:19:24.917775 2699 log.cc:1196] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-0/wal/wals/6ed51afc59df441a8b31d95456d61108
I20260924 00:19:24.918080 2699 ts_tablet_manager.cc:1939] T 6ed51afc59df441a8b31d95456d61108 P 86b78a425f984265aa22d4a1917f23aa: Deleting consensus metadata
I20260924 00:19:24.918406 2700 tablet_replica.cc:330] T 6ed51afc59df441a8b31d95456d61108 P a4cf5c98df9a431e96a03b29534699cf: stopping tablet replica
I20260924 00:19:24.918474 2109 catalog_manager.cc:4841] TS 86b78a425f984265aa22d4a1917f23aa (127.0.98.1:40309): tablet 6ed51afc59df441a8b31d95456d61108 (table test-table [id=d12702723e90476eaa39054f91b2df86]) successfully deleted
I20260924 00:19:24.918531 2700 raft_consensus.cc:2229] T 6ed51afc59df441a8b31d95456d61108 P a4cf5c98df9a431e96a03b29534699cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:24.918584 2700 raft_consensus.cc:2258] T 6ed51afc59df441a8b31d95456d61108 P a4cf5c98df9a431e96a03b29534699cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:24.918876 2700 ts_tablet_manager.cc:1905] T 6ed51afc59df441a8b31d95456d61108 P a4cf5c98df9a431e96a03b29534699cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:24.919325 2698 ts_tablet_manager.cc:1918] T 6ed51afc59df441a8b31d95456d61108 P f59aa39d2fa147cc8faf18842d35958e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 00:19:24.919404 2698 log.cc:1196] T 6ed51afc59df441a8b31d95456d61108 P f59aa39d2fa147cc8faf18842d35958e: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-1/wal/wals/6ed51afc59df441a8b31d95456d61108
I20260924 00:19:24.919677 2698 ts_tablet_manager.cc:1939] T 6ed51afc59df441a8b31d95456d61108 P f59aa39d2fa147cc8faf18842d35958e: Deleting consensus metadata
I20260924 00:19:24.920087 2107 catalog_manager.cc:4841] TS f59aa39d2fa147cc8faf18842d35958e (127.0.98.2:32917): tablet 6ed51afc59df441a8b31d95456d61108 (table test-table [id=d12702723e90476eaa39054f91b2df86]) successfully deleted
I20260924 00:19:24.920828 2700 ts_tablet_manager.cc:1918] T 6ed51afc59df441a8b31d95456d61108 P a4cf5c98df9a431e96a03b29534699cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 00:19:24.920897 2700 log.cc:1196] T 6ed51afc59df441a8b31d95456d61108 P a4cf5c98df9a431e96a03b29534699cf: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/6ed51afc59df441a8b31d95456d61108
I20260924 00:19:24.921165 2700 ts_tablet_manager.cc:1939] T 6ed51afc59df441a8b31d95456d61108 P a4cf5c98df9a431e96a03b29534699cf: Deleting consensus metadata
I20260924 00:19:24.921554 2107 catalog_manager.cc:4841] TS a4cf5c98df9a431e96a03b29534699cf (127.0.98.3:39273): tablet 6ed51afc59df441a8b31d95456d61108 (table test-table [id=d12702723e90476eaa39054f91b2df86]) successfully deleted
Sep 24 00:19:24 dist-test-slave-777x krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209164, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 00:19:24 dist-test-slave-777x krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209164, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:24.948523 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:24.937964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58082 (local address 127.0.98.62:45675)
0924 00:19:24.938119 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:24.938123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:24.938195 (+ 72us) server_negotiation.cc:407] Connection header received
0924 00:19:24.938299 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:24.938300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:24.938315 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:24.938425 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:24.939235 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:24.940166 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:24.941423 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:24.941794 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:24.941864 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:24.945132 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:24.945134 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:24.945135 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:24.945143 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:24.946970 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:24.947396 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:24.947398 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:24.947398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:24.947469 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:24.947702 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:24.947703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:24.947703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:24.947834 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:24.947953 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:24.948305 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 00:19:24.948408 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260924 00:19:24.950361 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:19:24.950565 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:19:24.951992 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 9bd24858350c4d8abb9cceeffbd8f81f (DEFAULT_TABLE table=test-table [id=92cb3c4a65be4bd8b49acff235469658]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:24.952050 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 9bd24858350c4d8abb9cceeffbd8f81f (DEFAULT_TABLE table=test-table [id=92cb3c4a65be4bd8b49acff235469658]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:24.952103 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 9bd24858350c4d8abb9cceeffbd8f81f (DEFAULT_TABLE table=test-table [id=92cb3c4a65be4bd8b49acff235469658]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:24.952139 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bd24858350c4d8abb9cceeffbd8f81f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:24.952217 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bd24858350c4d8abb9cceeffbd8f81f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:24.952241 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bd24858350c4d8abb9cceeffbd8f81f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:24.953563 2709 tablet_bootstrap.cc:492] T 9bd24858350c4d8abb9cceeffbd8f81f P 86b78a425f984265aa22d4a1917f23aa: Bootstrap starting.
I20260924 00:19:24.953740 2710 tablet_bootstrap.cc:492] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e: Bootstrap starting.
I20260924 00:19:24.953989 2709 tablet_bootstrap.cc:654] T 9bd24858350c4d8abb9cceeffbd8f81f P 86b78a425f984265aa22d4a1917f23aa: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:24.954101 2711 tablet_bootstrap.cc:492] T 9bd24858350c4d8abb9cceeffbd8f81f P a4cf5c98df9a431e96a03b29534699cf: Bootstrap starting.
I20260924 00:19:24.954123 2710 tablet_bootstrap.cc:654] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:24.954537 2711 tablet_bootstrap.cc:654] T 9bd24858350c4d8abb9cceeffbd8f81f P a4cf5c98df9a431e96a03b29534699cf: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:24.954697 2709 tablet_bootstrap.cc:492] T 9bd24858350c4d8abb9cceeffbd8f81f P 86b78a425f984265aa22d4a1917f23aa: No bootstrap required, opened a new log
I20260924 00:19:24.954704 2710 tablet_bootstrap.cc:492] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e: No bootstrap required, opened a new log
I20260924 00:19:24.954748 2710 ts_tablet_manager.cc:1397] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:24.954838 2709 ts_tablet_manager.cc:1397] T 9bd24858350c4d8abb9cceeffbd8f81f P 86b78a425f984265aa22d4a1917f23aa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:24.954890 2710 raft_consensus.cc:348] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:24.954975 2710 raft_consensus.cc:374] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:24.955008 2710 raft_consensus.cc:729] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f59aa39d2fa147cc8faf18842d35958e, State: Initialized, Role: FOLLOWER
I20260924 00:19:24.955040 2709 raft_consensus.cc:348] T 9bd24858350c4d8abb9cceeffbd8f81f P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:24.955081 2710 consensus_queue.cc:260] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:24.955138 2709 raft_consensus.cc:374] T 9bd24858350c4d8abb9cceeffbd8f81f P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:24.955171 2709 raft_consensus.cc:729] T 9bd24858350c4d8abb9cceeffbd8f81f P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b78a425f984265aa22d4a1917f23aa, State: Initialized, Role: FOLLOWER
I20260924 00:19:24.955251 2710 ts_tablet_manager.cc:1428] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 00:19:24.955222 2709 consensus_queue.cc:260] T 9bd24858350c4d8abb9cceeffbd8f81f P 86b78a425f984265aa22d4a1917f23aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:24.955461 2709 ts_tablet_manager.cc:1428] T 9bd24858350c4d8abb9cceeffbd8f81f P 86b78a425f984265aa22d4a1917f23aa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 00:19:24.956122 2711 tablet_bootstrap.cc:492] T 9bd24858350c4d8abb9cceeffbd8f81f P a4cf5c98df9a431e96a03b29534699cf: No bootstrap required, opened a new log
I20260924 00:19:24.956261 2711 ts_tablet_manager.cc:1397] T 9bd24858350c4d8abb9cceeffbd8f81f P a4cf5c98df9a431e96a03b29534699cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:19:24.956480 2711 raft_consensus.cc:348] T 9bd24858350c4d8abb9cceeffbd8f81f P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:24.956593 2711 raft_consensus.cc:374] T 9bd24858350c4d8abb9cceeffbd8f81f P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:24.956624 2711 raft_consensus.cc:729] T 9bd24858350c4d8abb9cceeffbd8f81f P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cf5c98df9a431e96a03b29534699cf, State: Initialized, Role: FOLLOWER
I20260924 00:19:24.956691 2711 consensus_queue.cc:260] T 9bd24858350c4d8abb9cceeffbd8f81f P a4cf5c98df9a431e96a03b29534699cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:24.956871 2711 ts_tablet_manager.cc:1428] T 9bd24858350c4d8abb9cceeffbd8f81f P a4cf5c98df9a431e96a03b29534699cf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 00:19:24.960497 2715 raft_consensus.cc:482] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:19:24.960566 2715 raft_consensus.cc:504] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:24.961337 2715 leader_election.cc:288] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86b78a425f984265aa22d4a1917f23aa (127.0.98.1:40309), a4cf5c98df9a431e96a03b29534699cf (127.0.98.3:39273)
I20260924 00:19:24.965062 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:24.961046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40309 (local address 127.0.98.2:37175)
0924 00:19:24.961529 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:24.961537 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:24.961571 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:24.961737 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:24.961738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:24.961739 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:24.961792 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:24.961793 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:24.962689 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:24.962691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:24.963891 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:24.963893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:24.964818 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:24.964819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:24.964880 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:24.964884 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:19:24.964932 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 00:19:24.964974 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":2,"thread_start_us":100,"threads_started":1}
I20260924 00:19:24.965305 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:24.961242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37175 (local address 127.0.98.1:40309)
0924 00:19:24.961387 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:24.961389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:24.961613 (+ 224us) server_negotiation.cc:407] Connection header received
0924 00:19:24.961638 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:24.961639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:24.961648 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:24.961699 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:24.961853 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:24.962599 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:24.963985 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:24.964585 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:24.964630 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:24.964642 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:24.965200 (+ 558us) server_negotiation.cc:299] Negotiation successful
0924 00:19:24.965242 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260924 00:19:24.965499 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd24858350c4d8abb9cceeffbd8f81f" candidate_uuid: "f59aa39d2fa147cc8faf18842d35958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86b78a425f984265aa22d4a1917f23aa" is_pre_election: true
I20260924 00:19:24.965649 2280 raft_consensus.cc:2454] T 9bd24858350c4d8abb9cceeffbd8f81f P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f59aa39d2fa147cc8faf18842d35958e in term 0.
I20260924 00:19:24.965823 2353 leader_election.cc:302] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86b78a425f984265aa22d4a1917f23aa, f59aa39d2fa147cc8faf18842d35958e; no voters:
I20260924 00:19:24.965974 2715 raft_consensus.cc:2781] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:19:24.966012 2715 raft_consensus.cc:482] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:19:24.966040 2715 raft_consensus.cc:3037] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:24.966197 2717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:24.961046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39273 (local address 127.0.98.2:41587)
0924 00:19:24.961391 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:24.961403 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:24.961466 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:24.961679 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:24.961681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:24.961684 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:24.961771 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:24.961774 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:24.963254 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:24.963256 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:24.964684 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:24.964687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:24.965852 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:24.965853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:24.965929 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:24.965934 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:19:24.965982 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 00:19:24.966047 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":5,"thread_start_us":96,"threads_started":1}
I20260924 00:19:24.966476 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:24.961237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41587 (local address 127.0.98.3:39273)
0924 00:19:24.961366 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:24.961369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:24.961455 (+ 86us) server_negotiation.cc:407] Connection header received
0924 00:19:24.961522 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:24.961523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:24.961536 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:24.961619 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:24.961861 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:24.963144 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:24.964798 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:24.965735 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:24.965807 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:24.965818 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:24.966351 (+ 533us) server_negotiation.cc:299] Negotiation successful
0924 00:19:24.966397 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":75,"threads_started":1}
I20260924 00:19:24.966634 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd24858350c4d8abb9cceeffbd8f81f" candidate_uuid: "f59aa39d2fa147cc8faf18842d35958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cf5c98df9a431e96a03b29534699cf" is_pre_election: true
I20260924 00:19:24.966722 2559 raft_consensus.cc:2454] T 9bd24858350c4d8abb9cceeffbd8f81f P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f59aa39d2fa147cc8faf18842d35958e in term 0.
I20260924 00:19:24.966917 2715 raft_consensus.cc:504] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:24.967099 2715 leader_election.cc:288] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e [CANDIDATE]: Term 1 election: Requested vote from peers 86b78a425f984265aa22d4a1917f23aa (127.0.98.1:40309), a4cf5c98df9a431e96a03b29534699cf (127.0.98.3:39273)
I20260924 00:19:24.967183 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd24858350c4d8abb9cceeffbd8f81f" candidate_uuid: "f59aa39d2fa147cc8faf18842d35958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86b78a425f984265aa22d4a1917f23aa"
I20260924 00:19:24.967226 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd24858350c4d8abb9cceeffbd8f81f" candidate_uuid: "f59aa39d2fa147cc8faf18842d35958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4cf5c98df9a431e96a03b29534699cf"
I20260924 00:19:24.967274 2280 raft_consensus.cc:3037] T 9bd24858350c4d8abb9cceeffbd8f81f P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:24.967283 2559 raft_consensus.cc:3037] T 9bd24858350c4d8abb9cceeffbd8f81f P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:24.968043 2559 raft_consensus.cc:2454] T 9bd24858350c4d8abb9cceeffbd8f81f P a4cf5c98df9a431e96a03b29534699cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f59aa39d2fa147cc8faf18842d35958e in term 1.
I20260924 00:19:24.968154 2280 raft_consensus.cc:2454] T 9bd24858350c4d8abb9cceeffbd8f81f P 86b78a425f984265aa22d4a1917f23aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f59aa39d2fa147cc8faf18842d35958e in term 1.
I20260924 00:19:24.968271 2351 leader_election.cc:302] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cf5c98df9a431e96a03b29534699cf, f59aa39d2fa147cc8faf18842d35958e; no voters:
I20260924 00:19:24.968385 2715 raft_consensus.cc:2781] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:24.968550 2715 raft_consensus.cc:686] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e [term 1 LEADER]: Becoming Leader. State: Replica: f59aa39d2fa147cc8faf18842d35958e, State: Running, Role: LEADER
I20260924 00:19:24.968624 2715 consensus_queue.cc:237] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:24.969307 2122 catalog_manager.cc:5495] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e reported cstate change: term changed from 0 to 1, leader changed from <none> to f59aa39d2fa147cc8faf18842d35958e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f59aa39d2fa147cc8faf18842d35958e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } health_report { overall_health: UNKNOWN } } }
I20260924 00:19:24.979503 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:24.976866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33628 (local address 127.0.98.2:32917)
0924 00:19:24.977081 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:24.977082 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:24.977088 (+ 6us) server_negotiation.cc:407] Connection header received
0924 00:19:24.977120 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:24.977120 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:24.977137 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:24.977225 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:24.977384 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:24.977884 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:24.978649 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:24.978862 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:24.978950 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:24.979127 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:24.979242 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:24.979329 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:24.979374 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 00:19:24.979396 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":154,"threads_started":1}
I20260924 00:19:24.985191 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:24.982313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55349 (local address 127.0.98.1:40309)
0924 00:19:24.982372 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:24.982374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:24.982425 (+ 51us) server_negotiation.cc:407] Connection header received
0924 00:19:24.982538 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:24.982539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:24.982552 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:24.982635 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:24.982823 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:24.983373 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:24.984319 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:24.984543 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:24.984650 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:24.984869 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:24.984914 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:24.984999 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:24.985078 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 00:19:24.985103 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 00:19:24.985213 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:24.982142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40309 (local address 127.0.98.2:55349)
0924 00:19:24.982348 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:24.982363 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:24.982443 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:24.982641 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:24.982642 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:24.982644 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:19:24.982724 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:24.982726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:24.983483 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:24.983484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:24.984230 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:24.984233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:24.984667 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:24.984668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:24.984750 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:24.984766 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:19:24.985002 (+ 236us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:19:24.985005 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:19:24.985058 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 00:19:24.985112 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":62,"threads_started":1}
I20260924 00:19:24.987669 2559 raft_consensus.cc:1261] T 9bd24858350c4d8abb9cceeffbd8f81f P a4cf5c98df9a431e96a03b29534699cf [term 1 FOLLOWER]: Refusing update from remote peer f59aa39d2fa147cc8faf18842d35958e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:24.987716 2280 raft_consensus.cc:1261] T 9bd24858350c4d8abb9cceeffbd8f81f P 86b78a425f984265aa22d4a1917f23aa [term 1 FOLLOWER]: Refusing update from remote peer f59aa39d2fa147cc8faf18842d35958e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:24.987995 2721 consensus_queue.cc:1035] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:24.988131 2715 consensus_queue.cc:1035] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:24.995513 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:24.993019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40693 (local address 127.0.98.2:32917)
0924 00:19:24.993120 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:24.993123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:24.993180 (+ 57us) server_negotiation.cc:407] Connection header received
0924 00:19:24.993252 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:24.993253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:24.993263 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:24.993315 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:24.993440 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:24.993942 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:24.994696 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:24.995051 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:24.995110 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:24.995257 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:24.995294 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:24.995344 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:24.995409 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 00:19:24.995430 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 00:19:24.995522 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:24.992978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32917 (local address 127.0.98.1:40693)
0924 00:19:24.993118 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:24.993131 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:24.993194 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:24.993336 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:24.993337 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:24.993339 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:19:24.993387 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:24.993388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:24.994017 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:24.994018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:24.994612 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:24.994613 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:24.995111 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:24.995112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:24.995175 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:24.995181 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:19:24.995344 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:19:24.995346 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:19:24.995404 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 00:19:24.995450 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":88,"threads_started":1}
I20260924 00:19:25.002579 2123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:25.002725 2123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:25.003516 2123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6: Sending DeleteTablet for 3 replicas of tablet 9bd24858350c4d8abb9cceeffbd8f81f
I20260924 00:19:25.003845 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bd24858350c4d8abb9cceeffbd8f81f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52368
I20260924 00:19:25.003957 2699 tablet_replica.cc:330] T 9bd24858350c4d8abb9cceeffbd8f81f P 86b78a425f984265aa22d4a1917f23aa: stopping tablet replica
I20260924 00:19:25.003937 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bd24858350c4d8abb9cceeffbd8f81f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33626
I20260924 00:19:25.004004 2699 raft_consensus.cc:2229] T 9bd24858350c4d8abb9cceeffbd8f81f P 86b78a425f984265aa22d4a1917f23aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:25.003991 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bd24858350c4d8abb9cceeffbd8f81f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34760
I20260924 00:19:25.004045 2699 raft_consensus.cc:2258] T 9bd24858350c4d8abb9cceeffbd8f81f P 86b78a425f984265aa22d4a1917f23aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:25.004122 2698 tablet_replica.cc:330] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e: stopping tablet replica
I20260924 00:19:25.004235 2700 tablet_replica.cc:330] T 9bd24858350c4d8abb9cceeffbd8f81f P a4cf5c98df9a431e96a03b29534699cf: stopping tablet replica
I20260924 00:19:25.004302 2700 raft_consensus.cc:2229] T 9bd24858350c4d8abb9cceeffbd8f81f P a4cf5c98df9a431e96a03b29534699cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:25.004370 2700 raft_consensus.cc:2258] T 9bd24858350c4d8abb9cceeffbd8f81f P a4cf5c98df9a431e96a03b29534699cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:25.004528 2698 raft_consensus.cc:2229] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:19:25.004576 2699 ts_tablet_manager.cc:1905] T 9bd24858350c4d8abb9cceeffbd8f81f P 86b78a425f984265aa22d4a1917f23aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:25.004637 2698 raft_consensus.cc:2258] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:25.004647 2700 ts_tablet_manager.cc:1905] T 9bd24858350c4d8abb9cceeffbd8f81f P a4cf5c98df9a431e96a03b29534699cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:25.005046 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:19:25.005214 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:19:25.006429 2698 ts_tablet_manager.cc:1905] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:25.006596 2260 tablet_service.cc:1467] Processing CreateTablet for tablet dd140d85d7bb4200a50b027da006ae18 (DEFAULT_TABLE table=test-table [id=d61d140356b04777a01ca30dc37abd43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:25.006742 2400 tablet_service.cc:1467] Processing CreateTablet for tablet dd140d85d7bb4200a50b027da006ae18 (DEFAULT_TABLE table=test-table [id=d61d140356b04777a01ca30dc37abd43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:25.006759 2540 tablet_service.cc:1467] Processing CreateTablet for tablet dd140d85d7bb4200a50b027da006ae18 (DEFAULT_TABLE table=test-table [id=d61d140356b04777a01ca30dc37abd43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:25.006834 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd140d85d7bb4200a50b027da006ae18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:25.007984 2698 ts_tablet_manager.cc:1918] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:19:25.008016 2710 tablet_bootstrap.cc:492] T dd140d85d7bb4200a50b027da006ae18 P f59aa39d2fa147cc8faf18842d35958e: Bootstrap starting.
I20260924 00:19:25.008036 2698 log.cc:1196] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-1/wal/wals/9bd24858350c4d8abb9cceeffbd8f81f
I20260924 00:19:25.008296 2698 ts_tablet_manager.cc:1939] T 9bd24858350c4d8abb9cceeffbd8f81f P f59aa39d2fa147cc8faf18842d35958e: Deleting consensus metadata
I20260924 00:19:25.008353 2700 ts_tablet_manager.cc:1918] T 9bd24858350c4d8abb9cceeffbd8f81f P a4cf5c98df9a431e96a03b29534699cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:19:25.008419 2700 log.cc:1196] T 9bd24858350c4d8abb9cceeffbd8f81f P a4cf5c98df9a431e96a03b29534699cf: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/9bd24858350c4d8abb9cceeffbd8f81f
I20260924 00:19:25.008430 2710 tablet_bootstrap.cc:654] T dd140d85d7bb4200a50b027da006ae18 P f59aa39d2fa147cc8faf18842d35958e: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:25.008713 2700 ts_tablet_manager.cc:1939] T 9bd24858350c4d8abb9cceeffbd8f81f P a4cf5c98df9a431e96a03b29534699cf: Deleting consensus metadata
I20260924 00:19:25.008952 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd140d85d7bb4200a50b027da006ae18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:25.009145 2710 tablet_bootstrap.cc:492] T dd140d85d7bb4200a50b027da006ae18 P f59aa39d2fa147cc8faf18842d35958e: No bootstrap required, opened a new log
I20260924 00:19:25.009212 2710 ts_tablet_manager.cc:1397] T dd140d85d7bb4200a50b027da006ae18 P f59aa39d2fa147cc8faf18842d35958e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:25.009369 2710 raft_consensus.cc:348] T dd140d85d7bb4200a50b027da006ae18 P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:25.009450 2710 raft_consensus.cc:374] T dd140d85d7bb4200a50b027da006ae18 P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:25.009478 2710 raft_consensus.cc:729] T dd140d85d7bb4200a50b027da006ae18 P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f59aa39d2fa147cc8faf18842d35958e, State: Initialized, Role: FOLLOWER
I20260924 00:19:25.009518 2710 consensus_queue.cc:260] T dd140d85d7bb4200a50b027da006ae18 P f59aa39d2fa147cc8faf18842d35958e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:25.009641 2710 ts_tablet_manager.cc:1428] T dd140d85d7bb4200a50b027da006ae18 P f59aa39d2fa147cc8faf18842d35958e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 00:19:25.010406 2711 tablet_bootstrap.cc:492] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf: Bootstrap starting.
I20260924 00:19:25.010717 2107 catalog_manager.cc:4841] TS f59aa39d2fa147cc8faf18842d35958e (127.0.98.2:32917): tablet 9bd24858350c4d8abb9cceeffbd8f81f (table test-table [id=92cb3c4a65be4bd8b49acff235469658]) successfully deleted
I20260924 00:19:25.010879 2711 tablet_bootstrap.cc:654] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:25.011461 2699 ts_tablet_manager.cc:1918] T 9bd24858350c4d8abb9cceeffbd8f81f P 86b78a425f984265aa22d4a1917f23aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:19:25.011557 2699 log.cc:1196] T 9bd24858350c4d8abb9cceeffbd8f81f P 86b78a425f984265aa22d4a1917f23aa: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-0/wal/wals/9bd24858350c4d8abb9cceeffbd8f81f
I20260924 00:19:25.011655 2711 tablet_bootstrap.cc:492] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf: No bootstrap required, opened a new log
I20260924 00:19:25.011709 2711 ts_tablet_manager.cc:1397] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:25.011828 2699 ts_tablet_manager.cc:1939] T 9bd24858350c4d8abb9cceeffbd8f81f P 86b78a425f984265aa22d4a1917f23aa: Deleting consensus metadata
I20260924 00:19:25.011855 2711 raft_consensus.cc:348] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:25.011950 2711 raft_consensus.cc:374] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:25.011976 2711 raft_consensus.cc:729] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cf5c98df9a431e96a03b29534699cf, State: Initialized, Role: FOLLOWER
I20260924 00:19:25.012046 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd140d85d7bb4200a50b027da006ae18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:25.012037 2711 consensus_queue.cc:260] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:25.012128 2711 ts_tablet_manager.cc:1428] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 00:19:25.013713 2709 tablet_bootstrap.cc:492] T dd140d85d7bb4200a50b027da006ae18 P 86b78a425f984265aa22d4a1917f23aa: Bootstrap starting.
I20260924 00:19:25.014220 2709 tablet_bootstrap.cc:654] T dd140d85d7bb4200a50b027da006ae18 P 86b78a425f984265aa22d4a1917f23aa: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:25.014540 2107 catalog_manager.cc:4841] TS a4cf5c98df9a431e96a03b29534699cf (127.0.98.3:39273): tablet 9bd24858350c4d8abb9cceeffbd8f81f (table test-table [id=92cb3c4a65be4bd8b49acff235469658]) successfully deleted
I20260924 00:19:25.014789 2109 catalog_manager.cc:4841] TS 86b78a425f984265aa22d4a1917f23aa (127.0.98.1:40309): tablet 9bd24858350c4d8abb9cceeffbd8f81f (table test-table [id=92cb3c4a65be4bd8b49acff235469658]) successfully deleted
I20260924 00:19:25.014948 2709 tablet_bootstrap.cc:492] T dd140d85d7bb4200a50b027da006ae18 P 86b78a425f984265aa22d4a1917f23aa: No bootstrap required, opened a new log
I20260924 00:19:25.015004 2709 ts_tablet_manager.cc:1397] T dd140d85d7bb4200a50b027da006ae18 P 86b78a425f984265aa22d4a1917f23aa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:25.015131 2709 raft_consensus.cc:348] T dd140d85d7bb4200a50b027da006ae18 P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:25.015201 2709 raft_consensus.cc:374] T dd140d85d7bb4200a50b027da006ae18 P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:25.015216 2709 raft_consensus.cc:729] T dd140d85d7bb4200a50b027da006ae18 P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b78a425f984265aa22d4a1917f23aa, State: Initialized, Role: FOLLOWER
I20260924 00:19:25.015275 2709 consensus_queue.cc:260] T dd140d85d7bb4200a50b027da006ae18 P 86b78a425f984265aa22d4a1917f23aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:25.015372 2709 ts_tablet_manager.cc:1428] T dd140d85d7bb4200a50b027da006ae18 P 86b78a425f984265aa22d4a1917f23aa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:25.135968 2716 raft_consensus.cc:482] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:19:25.136102 2716 raft_consensus.cc:504] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:25.136560 2716 leader_election.cc:288] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86b78a425f984265aa22d4a1917f23aa (127.0.98.1:40309), f59aa39d2fa147cc8faf18842d35958e (127.0.98.2:32917)
I20260924 00:19:25.139797 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:25.136836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47355 (local address 127.0.98.1:40309)
0924 00:19:25.136905 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:25.136907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:25.136974 (+ 67us) server_negotiation.cc:407] Connection header received
0924 00:19:25.137057 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:25.137058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:25.137067 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:25.137113 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:25.137293 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:25.137962 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:25.139005 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:25.139387 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:25.139505 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:25.139519 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:25.139678 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 00:19:25.139718 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 00:19:25.139817 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:25.136768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40309 (local address 127.0.98.3:47355)
0924 00:19:25.136906 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:25.136918 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:25.136993 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:25.137126 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:25.137127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:25.137129 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:25.137211 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:25.137213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:25.138078 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:25.138079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:25.138897 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:25.138900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:25.139518 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:25.139519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:25.139592 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:25.139596 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:19:25.139672 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 00:19:25.139718 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":15,"thread_start_us":60,"threads_started":1}
I20260924 00:19:25.139878 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:25.136888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52083 (local address 127.0.98.2:32917)
0924 00:19:25.136938 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:25.136940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:25.137082 (+ 142us) server_negotiation.cc:407] Connection header received
0924 00:19:25.137158 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:25.137160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:25.137172 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:25.137236 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:25.137417 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:25.138044 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:25.139052 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:25.139470 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:25.139564 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:25.139577 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:25.139758 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 00:19:25.139799 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:19:25.139935 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:25.136788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32917 (local address 127.0.98.3:52083)
0924 00:19:25.137005 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:25.137014 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:25.137097 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:25.137255 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:25.137256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:25.137259 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:25.137340 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:25.137342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:25.138135 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:25.138136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:25.138944 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:25.138946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:25.139566 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:25.139567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:25.139664 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:25.139668 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:19:25.139738 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 00:19:25.139786 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":122,"threads_started":1}
I20260924 00:19:25.140324 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd140d85d7bb4200a50b027da006ae18" candidate_uuid: "a4cf5c98df9a431e96a03b29534699cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86b78a425f984265aa22d4a1917f23aa" is_pre_election: true
I20260924 00:19:25.140360 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd140d85d7bb4200a50b027da006ae18" candidate_uuid: "a4cf5c98df9a431e96a03b29534699cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f59aa39d2fa147cc8faf18842d35958e" is_pre_election: true
I20260924 00:19:25.140482 2280 raft_consensus.cc:2454] T dd140d85d7bb4200a50b027da006ae18 P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cf5c98df9a431e96a03b29534699cf in term 0.
I20260924 00:19:25.140479 2420 raft_consensus.cc:2454] T dd140d85d7bb4200a50b027da006ae18 P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4cf5c98df9a431e96a03b29534699cf in term 0.
I20260924 00:19:25.140671 2493 leader_election.cc:302] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86b78a425f984265aa22d4a1917f23aa, a4cf5c98df9a431e96a03b29534699cf; no voters:
I20260924 00:19:25.140873 2716 raft_consensus.cc:2781] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:19:25.140923 2716 raft_consensus.cc:482] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:19:25.141073 2716 raft_consensus.cc:3037] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:25.141804 2716 raft_consensus.cc:504] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:25.141983 2716 leader_election.cc:288] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf [CANDIDATE]: Term 1 election: Requested vote from peers 86b78a425f984265aa22d4a1917f23aa (127.0.98.1:40309), f59aa39d2fa147cc8faf18842d35958e (127.0.98.2:32917)
I20260924 00:19:25.142201 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd140d85d7bb4200a50b027da006ae18" candidate_uuid: "a4cf5c98df9a431e96a03b29534699cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f59aa39d2fa147cc8faf18842d35958e"
I20260924 00:19:25.142282 2420 raft_consensus.cc:3037] T dd140d85d7bb4200a50b027da006ae18 P f59aa39d2fa147cc8faf18842d35958e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:25.142274 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd140d85d7bb4200a50b027da006ae18" candidate_uuid: "a4cf5c98df9a431e96a03b29534699cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86b78a425f984265aa22d4a1917f23aa"
I20260924 00:19:25.142390 2280 raft_consensus.cc:3037] T dd140d85d7bb4200a50b027da006ae18 P 86b78a425f984265aa22d4a1917f23aa [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:25.142884 2420 raft_consensus.cc:2454] T dd140d85d7bb4200a50b027da006ae18 P f59aa39d2fa147cc8faf18842d35958e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cf5c98df9a431e96a03b29534699cf in term 1.
I20260924 00:19:25.143100 2280 raft_consensus.cc:2454] T dd140d85d7bb4200a50b027da006ae18 P 86b78a425f984265aa22d4a1917f23aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4cf5c98df9a431e96a03b29534699cf in term 1.
I20260924 00:19:25.143158 2491 leader_election.cc:302] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4cf5c98df9a431e96a03b29534699cf, f59aa39d2fa147cc8faf18842d35958e; no voters:
I20260924 00:19:25.143291 2716 raft_consensus.cc:2781] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:25.143496 2716 raft_consensus.cc:686] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf [term 1 LEADER]: Becoming Leader. State: Replica: a4cf5c98df9a431e96a03b29534699cf, State: Running, Role: LEADER
I20260924 00:19:25.143582 2716 consensus_queue.cc:237] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } }
I20260924 00:19:25.144479 2123 catalog_manager.cc:5495] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf reported cstate change: term changed from 0 to 1, leader changed from <none> to a4cf5c98df9a431e96a03b29534699cf (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a4cf5c98df9a431e96a03b29534699cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4cf5c98df9a431e96a03b29534699cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39273 } health_report { overall_health: HEALTHY } } }
I20260924 00:19:25.163851 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:25.161308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34770 (local address 127.0.98.3:39273)
0924 00:19:25.161376 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:25.161378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:25.161406 (+ 28us) server_negotiation.cc:407] Connection header received
0924 00:19:25.161449 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:25.161450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:25.161484 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:25.161578 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:25.161705 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:25.162201 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:25.162983 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:25.163195 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:25.163289 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:25.163476 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:25.163594 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:25.163685 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:25.163730 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 00:19:25.163754 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 00:19:25.169292 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:25.166694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40309 (local address 127.0.98.3:57515)
0924 00:19:25.166846 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:25.166858 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:25.166911 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:25.167113 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:25.167114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:25.167116 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:19:25.167177 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:25.167178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:25.167812 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:25.167813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:25.168489 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:25.168491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:25.168841 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:25.168842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:25.168906 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:25.168925 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:19:25.169113 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:19:25.169115 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:19:25.169162 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 00:19:25.169204 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260924 00:19:25.169271 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:25.166856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57515 (local address 127.0.98.1:40309)
0924 00:19:25.167006 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:25.167008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:25.167015 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:19:25.167035 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:25.167036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:25.167044 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:25.167109 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:25.167261 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:25.167731 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:25.168573 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:25.168768 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:25.168843 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:25.169006 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:25.169042 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:25.169099 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:25.169167 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 00:19:25.169190 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260924 00:19:25.171686 2420 raft_consensus.cc:1261] T dd140d85d7bb4200a50b027da006ae18 P f59aa39d2fa147cc8faf18842d35958e [term 1 FOLLOWER]: Refusing update from remote peer a4cf5c98df9a431e96a03b29534699cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:25.171695 2280 raft_consensus.cc:1261] T dd140d85d7bb4200a50b027da006ae18 P 86b78a425f984265aa22d4a1917f23aa [term 1 FOLLOWER]: Refusing update from remote peer a4cf5c98df9a431e96a03b29534699cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:25.171962 2739 consensus_queue.cc:1035] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f59aa39d2fa147cc8faf18842d35958e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:25.172086 2716 consensus_queue.cc:1035] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "86b78a425f984265aa22d4a1917f23aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:25.179406 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:25.176780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39273 (local address 127.0.98.1:45227)
0924 00:19:25.176865 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:25.176876 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:25.176929 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:25.177059 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:25.177060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:25.177061 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:19:25.177108 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:25.177110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:25.177853 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:25.177853 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:25.178481 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:25.178483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:25.179068 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:25.179069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:25.179127 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:25.179140 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:19:25.179289 (+ 149us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:19:25.179290 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 00:19:25.179319 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 00:19:25.179353 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 00:19:25.179400 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:25.176814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45227 (local address 127.0.98.3:39273)
0924 00:19:25.176872 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:25.176874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:25.176921 (+ 47us) server_negotiation.cc:407] Connection header received
0924 00:19:25.176969 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:25.176970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:25.176978 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:25.177055 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:25.177192 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:25.177775 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:25.178597 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:25.178993 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:25.179070 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:25.179211 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:25.179241 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:25.179288 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:25.179322 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 00:19:25.179341 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 00:19:25.180004 2682 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 00:19:26.706457 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:26.706573 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:26.707583 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 59eaae68dcf8488681c14e44fdf65dd6: Sending DeleteTablet for 3 replicas of tablet dd140d85d7bb4200a50b027da006ae18
I20260924 00:19:26.707894 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet dd140d85d7bb4200a50b027da006ae18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52368
I20260924 00:19:26.707934 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet dd140d85d7bb4200a50b027da006ae18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33626
I20260924 00:19:26.708108 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet dd140d85d7bb4200a50b027da006ae18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34760
I20260924 00:19:26.708151 2769 tablet_replica.cc:330] T dd140d85d7bb4200a50b027da006ae18 P f59aa39d2fa147cc8faf18842d35958e: stopping tablet replica
I20260924 00:19:26.708264 2769 raft_consensus.cc:2229] T dd140d85d7bb4200a50b027da006ae18 P f59aa39d2fa147cc8faf18842d35958e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:26.708335 2771 tablet_replica.cc:330] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf: stopping tablet replica
I20260924 00:19:26.708352 2769 raft_consensus.cc:2258] T dd140d85d7bb4200a50b027da006ae18 P f59aa39d2fa147cc8faf18842d35958e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:26.708398 2771 raft_consensus.cc:2229] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:19:26.708400 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 2195
I20260924 00:19:26.709054 2771 raft_consensus.cc:2258] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:26.709368 2771 ts_tablet_manager.cc:1905] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:26.710891 2771 ts_tablet_manager.cc:1918] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 00:19:26.710968 2771 log.cc:1196] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/dd140d85d7bb4200a50b027da006ae18
I20260924 00:19:26.711292 2771 ts_tablet_manager.cc:1939] T dd140d85d7bb4200a50b027da006ae18 P a4cf5c98df9a431e96a03b29534699cf: Deleting consensus metadata
I20260924 00:19:26.711335 2769 ts_tablet_manager.cc:1905] T dd140d85d7bb4200a50b027da006ae18 P f59aa39d2fa147cc8faf18842d35958e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:26.711676 2107 catalog_manager.cc:4841] TS a4cf5c98df9a431e96a03b29534699cf (127.0.98.3:39273): tablet dd140d85d7bb4200a50b027da006ae18 (table test-table [id=d61d140356b04777a01ca30dc37abd43]) successfully deleted
I20260924 00:19:26.713523 2769 ts_tablet_manager.cc:1918] T dd140d85d7bb4200a50b027da006ae18 P f59aa39d2fa147cc8faf18842d35958e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 00:19:26.713598 2769 log.cc:1196] T dd140d85d7bb4200a50b027da006ae18 P f59aa39d2fa147cc8faf18842d35958e: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790209154318147-392-0/minicluster-data/ts-1/wal/wals/dd140d85d7bb4200a50b027da006ae18
I20260924 00:19:26.713833 2769 ts_tablet_manager.cc:1939] T dd140d85d7bb4200a50b027da006ae18 P f59aa39d2fa147cc8faf18842d35958e: Deleting consensus metadata
I20260924 00:19:26.714512 2107 catalog_manager.cc:4841] TS f59aa39d2fa147cc8faf18842d35958e (127.0.98.2:32917): tablet dd140d85d7bb4200a50b027da006ae18 (table test-table [id=d61d140356b04777a01ca30dc37abd43]) successfully deleted
W20260924 00:19:26.716218 2109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40309 (error 108)
I20260924 00:19:26.716928 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 2335
I20260924 00:19:26.716962 2773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:26.716692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40309 (local address 127.0.0.1:52388)
0924 00:19:26.716860 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:26.716914 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
W20260924 00:19:26.717918 2109 catalog_manager.cc:4584] TS 86b78a425f984265aa22d4a1917f23aa (127.0.98.1:40309): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dd140d85d7bb4200a50b027da006ae18: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40309: connect: Connection refused (error 111)
I20260924 00:19:26.723119 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 2475
I20260924 00:19:26.728943 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 2091
2026-09-24T00:19:26Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5751 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:19:26 dist-test-slave-777x krb5kdc[2776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:19:26 dist-test-slave-777x krb5kdc[2776](info): set up 2 sockets
Sep 24 00:19:26 dist-test-slave-777x krb5kdc[2776](info): commencing operation
krb5kdc: starting...
W20260924 00:19:27.839926 392 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:19:27 dist-test-slave-777x krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:19:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:19:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:19:27.965574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40497
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:28.046985 2789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:28.047197 2789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:28.047219 2789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:28.048448 2789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:19:28.048498 2789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:28.048513 2789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:28.048527 2789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:19:28.048539 2789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:19:28.050065 2789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:28.050326 2789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:28.050580 2789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:28.055589 2797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:28.055625 2795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:28.055593 2794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:28.055954 2789 server_base.cc:1034] running on GCE node
I20260924 00:19:28.056270 2789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:28.056568 2789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:28.057713 2789 hybrid_clock.cc:648] HybridClock initialized: now 1790209168057688 us; error 39 us; skew 500 ppm
Sep 24 00:19:28 dist-test-slave-777x krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:28.060621 2789 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:19:28.061046 2789 webserver.cc:460] Webserver started at http://127.0.98.62:41217/ using document root <none> and password file <none>
I20260924 00:19:28.061250 2789 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:28.061295 2789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:28.061389 2789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:28.062561 2789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "d91b72fcca024332bc9a35989901a72a"
format_stamp: "Formatted at 2026-09-24 00:19:28 on dist-test-slave-777x"
server_key: "d82ec7337162943c6850d93eb2f0e8f4"
server_key_iv: "01ef24487b406a39b30ec448a6025d27"
server_key_version: "encryptionkey@0"
I20260924 00:19:28.062884 2789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "d91b72fcca024332bc9a35989901a72a"
format_stamp: "Formatted at 2026-09-24 00:19:28 on dist-test-slave-777x"
server_key: "d82ec7337162943c6850d93eb2f0e8f4"
server_key_iv: "01ef24487b406a39b30ec448a6025d27"
server_key_version: "encryptionkey@0"
I20260924 00:19:28.064409 2789 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 00:19:28.065246 2804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:28.065460 2789 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:28.065560 2789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "d91b72fcca024332bc9a35989901a72a"
format_stamp: "Formatted at 2026-09-24 00:19:28 on dist-test-slave-777x"
server_key: "d82ec7337162943c6850d93eb2f0e8f4"
server_key_iv: "01ef24487b406a39b30ec448a6025d27"
server_key_version: "encryptionkey@0"
I20260924 00:19:28.065619 2789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:28.077755 2789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:28.079701 2789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:28.079818 2789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:28.083753 2789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40497
I20260924 00:19:28.083786 2856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40497 every 8 connection(s)
I20260924 00:19:28.084336 2789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:19:28.085067 2857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:28.087356 2857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a: Bootstrap starting.
I20260924 00:19:28.087954 2857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:28.088272 2857 log.cc:824] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:28.088986 2857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a: No bootstrap required, opened a new log
I20260924 00:19:28.090588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 2789
I20260924 00:19:28.090689 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:19:28.090854 392 external_mini_cluster.cc:1428] Setting key f204ed195b48be16427af31498dac2de
I20260924 00:19:28.090829 2857 raft_consensus.cc:348] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d91b72fcca024332bc9a35989901a72a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40497 } }
I20260924 00:19:28.091004 2857 raft_consensus.cc:374] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:28.091043 2857 raft_consensus.cc:729] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d91b72fcca024332bc9a35989901a72a, State: Initialized, Role: FOLLOWER
I20260924 00:19:28.091202 2857 consensus_queue.cc:260] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d91b72fcca024332bc9a35989901a72a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40497 } }
I20260924 00:19:28.091295 2857 raft_consensus.cc:388] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:19:28.091332 2857 raft_consensus.cc:482] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:19:28.091408 2857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:28.092155 2857 raft_consensus.cc:504] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d91b72fcca024332bc9a35989901a72a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40497 } }
I20260924 00:19:28.092330 2857 leader_election.cc:302] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d91b72fcca024332bc9a35989901a72a; no voters:
I20260924 00:19:28.092492 2857 leader_election.cc:288] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:19:28.092783 2857 sys_catalog.cc:564] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:19:28.092515 2862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:28.093429 2862 raft_consensus.cc:686] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a [term 1 LEADER]: Becoming Leader. State: Replica: d91b72fcca024332bc9a35989901a72a, State: Running, Role: LEADER
I20260924 00:19:28.093545 2862 consensus_queue.cc:237] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d91b72fcca024332bc9a35989901a72a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40497 } }
I20260924 00:19:28.093895 2862 sys_catalog.cc:455] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d91b72fcca024332bc9a35989901a72a. Latest consensus state: current_term: 1 leader_uuid: "d91b72fcca024332bc9a35989901a72a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d91b72fcca024332bc9a35989901a72a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40497 } } }
I20260924 00:19:28.094000 2862 sys_catalog.cc:458] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:28.094127 2863 sys_catalog.cc:455] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d91b72fcca024332bc9a35989901a72a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d91b72fcca024332bc9a35989901a72a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40497 } } }
I20260924 00:19:28.094388 2863 sys_catalog.cc:458] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:28.094270 2876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:19:28.094939 2876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:19:28.096704 2876 catalog_manager.cc:1347] Generated new cluster ID: 2b5d1b18e179457faa71c32d5b6a12a8
I20260924 00:19:28.096760 2876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:19:28 dist-test-slave-777x krb5kdc[2776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:28.103214 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:28.091391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44532 (local address 127.0.98.62:40497)
0924 00:19:28.092079 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:28.092088 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:28.092115 (+ 27us) server_negotiation.cc:407] Connection header received
0924 00:19:28.092282 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:28.092295 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:28.092359 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:28.092521 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:28.093290 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:28.094920 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:28.096074 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:28.096848 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:28.096972 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:28.099559 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:28.099563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:28.099564 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:28.099578 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:28.101501 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:28.101924 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:28.101927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:28.101928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:28.102039 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:28.102269 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:28.102270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:28.102270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:28.102544 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:28.102737 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:28.102822 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 00:19:28.102992 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":537,"threads_started":1}
I20260924 00:19:28.111680 2876 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:19:28.112344 2876 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:19:28.118657 2876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d91b72fcca024332bc9a35989901a72a: Generated new TSK 0
I20260924 00:19:28.118857 2876 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:19:28.148727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40497
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:28.232303 2883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:28.232523 2883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:28.232548 2883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:28.233863 2883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:28.233912 2883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:28.233939 2883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:19:28.235447 2883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:28.235646 2883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:28.235827 2883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:28.241106 2891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:28.241142 2888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:28.241122 2889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:28.241549 2883 server_base.cc:1034] running on GCE node
I20260924 00:19:28.241777 2883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:28.241993 2883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:28.243222 2883 hybrid_clock.cc:648] HybridClock initialized: now 1790209168243211 us; error 33 us; skew 500 ppm
Sep 24 00:19:28 dist-test-slave-777x krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:28.246078 2883 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:19:28.246531 2883 webserver.cc:460] Webserver started at http://127.0.98.1:41431/ using document root <none> and password file <none>
I20260924 00:19:28.246783 2883 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:28.246852 2883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:28.247006 2883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:28.248096 2883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "4c33ef6ee10a4176a6cdbe2f2bccfa05"
format_stamp: "Formatted at 2026-09-24 00:19:28 on dist-test-slave-777x"
server_key: "83adbe475337a49ccae49c31a39ddab3"
server_key_iv: "8a3926ec1c4c94e3195bb16fd94c0513"
server_key_version: "encryptionkey@0"
I20260924 00:19:28.248522 2883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4c33ef6ee10a4176a6cdbe2f2bccfa05"
format_stamp: "Formatted at 2026-09-24 00:19:28 on dist-test-slave-777x"
server_key: "83adbe475337a49ccae49c31a39ddab3"
server_key_iv: "8a3926ec1c4c94e3195bb16fd94c0513"
server_key_version: "encryptionkey@0"
I20260924 00:19:28.249909 2883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:28.250761 2898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:28.250990 2883 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:28.251076 2883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "4c33ef6ee10a4176a6cdbe2f2bccfa05"
format_stamp: "Formatted at 2026-09-24 00:19:28 on dist-test-slave-777x"
server_key: "83adbe475337a49ccae49c31a39ddab3"
server_key_iv: "8a3926ec1c4c94e3195bb16fd94c0513"
server_key_version: "encryptionkey@0"
I20260924 00:19:28.251132 2883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:28.271768 2883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:28.274233 2883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:28.274364 2883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:28.274581 2883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:28.274967 2883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:28.275008 2883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:28.275038 2883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:28.275094 2883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:28.281512 2883 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40015
I20260924 00:19:28.281735 3011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40015 every 8 connection(s)
I20260924 00:19:28.281991 2883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:19:28.283913 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 2883
I20260924 00:19:28.284008 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:19:28.284271 392 external_mini_cluster.cc:1428] Setting key a987946d791d8eb6e0ceb61b89b7f099
Sep 24 00:19:28 dist-test-slave-777x krb5kdc[2776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 00:19:28.293340 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:28.282795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34171 (local address 127.0.98.62:40497)
0924 00:19:28.282886 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:28.282888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:28.283187 (+ 299us) server_negotiation.cc:407] Connection header received
0924 00:19:28.284202 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:28.284203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:28.284217 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:28.284269 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:28.285128 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:28.285633 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:28.287012 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:28.287234 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:28.287408 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:28.289962 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:28.289964 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:28.289965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:28.289976 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:28.291548 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:28.292085 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:28.292086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:28.292086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:28.292142 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:28.292551 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:28.292552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:28.292553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:28.292752 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:28.292824 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:28.293143 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 00:19:28.293234 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 00:19:28.293562 3014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:28.282747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40497 (local address 127.0.98.1:34171)
0924 00:19:28.282975 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:28.283015 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:28.284068 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:28.284343 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:28.284347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:28.284759 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:28.285023 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:28.285037 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:28.285731 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:28.285732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:28.286908 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:28.286911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:28.287423 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:28.287424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:28.287562 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:28.288163 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:28.288231 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:28.289842 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:28.291657 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:28.291664 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:28.291667 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:28.291931 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:28.292291 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:28.292292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:28.292292 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:28.292470 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:28.292814 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:28.292821 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:28.292987 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 00:19:28.293118 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 00:19:28.293301 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":101,"threads_started":1}
I20260924 00:19:28.294301 3012 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40497
I20260924 00:19:28.294428 3012 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:28.294636 3012 heartbeater.cc:511] Master 127.0.98.62:40497 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 00:19:28.295396 2821 ts_manager.cc:194] Registered new tserver with Master: 4c33ef6ee10a4176a6cdbe2f2bccfa05 (127.0.98.1:40015)
I20260924 00:19:28.296108 2821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:19:28.312788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40497
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:28.399330 3017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:28.399580 3017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:28.399608 3017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:28.400781 3017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:28.400825 3017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:28.400852 3017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:19:28.402338 3017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:28.402560 3017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:28.402752 3017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:28.408149 3022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:28.408185 3025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:28.408142 3023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:28.408298 3017 server_base.cc:1034] running on GCE node
I20260924 00:19:28.408564 3017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:28.408844 3017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:28.410029 3017 hybrid_clock.cc:648] HybridClock initialized: now 1790209168409996 us; error 54 us; skew 500 ppm
Sep 24 00:19:28 dist-test-slave-777x krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:28.413000 3017 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:19:28.413477 3017 webserver.cc:460] Webserver started at http://127.0.98.2:34937/ using document root <none> and password file <none>
I20260924 00:19:28.413727 3017 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:28.413798 3017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:28.413962 3017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:28.415277 3017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "0b678b49cc5a4674a30eb4fc8bcb3a8b"
format_stamp: "Formatted at 2026-09-24 00:19:28 on dist-test-slave-777x"
server_key: "7e3649e0e002b56fdfcf95bdfa56b9e2"
server_key_iv: "b2f28013b6050f18b8cb06177c60c8f5"
server_key_version: "encryptionkey@0"
I20260924 00:19:28.415745 3017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0b678b49cc5a4674a30eb4fc8bcb3a8b"
format_stamp: "Formatted at 2026-09-24 00:19:28 on dist-test-slave-777x"
server_key: "7e3649e0e002b56fdfcf95bdfa56b9e2"
server_key_iv: "b2f28013b6050f18b8cb06177c60c8f5"
server_key_version: "encryptionkey@0"
I20260924 00:19:28.417311 3017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260924 00:19:28.418143 3032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:28.418303 3017 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:28.418409 3017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "0b678b49cc5a4674a30eb4fc8bcb3a8b"
format_stamp: "Formatted at 2026-09-24 00:19:28 on dist-test-slave-777x"
server_key: "7e3649e0e002b56fdfcf95bdfa56b9e2"
server_key_iv: "b2f28013b6050f18b8cb06177c60c8f5"
server_key_version: "encryptionkey@0"
I20260924 00:19:28.418496 3017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:28.433185 3017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:28.435431 3017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:28.435619 3017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:28.435861 3017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:28.436260 3017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:28.436317 3017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:28.436362 3017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:28.436390 3017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:28.442754 3017 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46515
I20260924 00:19:28.442790 3145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46515 every 8 connection(s)
I20260924 00:19:28.443140 3017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:19:28.448801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 3017
I20260924 00:19:28.448899 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:19:28.449127 392 external_mini_cluster.cc:1428] Setting key 541c63caca289f45f5e5bf97d07c93c8
Sep 24 00:19:28 dist-test-slave-777x krb5kdc[2776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:28.453676 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:28.443950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59985 (local address 127.0.98.62:40497)
0924 00:19:28.444015 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:28.444017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:28.444235 (+ 218us) server_negotiation.cc:407] Connection header received
0924 00:19:28.445255 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:28.445256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:28.445267 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:28.445328 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:28.445963 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:28.446446 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:28.447555 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:28.447742 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:28.447801 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:28.450508 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:28.450510 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:28.450511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:28.450519 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:28.452097 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:28.452644 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:28.452645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:28.452645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:28.452698 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:28.452988 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:28.452988 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:28.452989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:28.453130 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:28.453182 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:28.453532 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 00:19:28.453601 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 00:19:28.454013 3148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:28.443931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40497 (local address 127.0.98.2:59985)
0924 00:19:28.444085 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:28.444128 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:28.445145 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:28.445397 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:28.445401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:28.445668 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:28.445877 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:28.445881 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:28.446547 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:28.446548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:28.447461 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:28.447464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:28.447838 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:28.447839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:28.447983 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:28.448513 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:28.448522 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:28.450401 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:28.452205 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:28.452214 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:28.452217 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:28.452559 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:28.452767 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:28.452768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:28.452768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:28.452916 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:28.453198 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:28.453204 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:28.453387 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 00:19:28.453499 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 00:19:28.453696 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260924 00:19:28.454519 3146 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40497
I20260924 00:19:28.454641 3146 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:28.454851 3146 heartbeater.cc:511] Master 127.0.98.62:40497 requested a full tablet report, sending...
I20260924 00:19:28.455308 2821 ts_manager.cc:194] Registered new tserver with Master: 0b678b49cc5a4674a30eb4fc8bcb3a8b (127.0.98.2:46515)
I20260924 00:19:28.455874 2821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59985
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:19:28.479406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40497
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:28.566202 3151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:28.566510 3151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:28.566555 3151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:28.568825 3151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:28.568893 3151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:28.568940 3151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:19:28.571192 3151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:28.571507 3151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:28.571754 3151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:19:28.578066 3151 server_base.cc:1034] running on GCE node
W20260924 00:19:28.578071 3156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:28.578008 3159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:28.578017 3157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:28.578506 3151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:28.578730 3151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:28.579908 3151 hybrid_clock.cc:648] HybridClock initialized: now 1790209168579895 us; error 29 us; skew 500 ppm
Sep 24 00:19:28 dist-test-slave-777x krb5kdc[2776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:28.582660 3151 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:19:28.583187 3151 webserver.cc:460] Webserver started at http://127.0.98.3:38095/ using document root <none> and password file <none>
I20260924 00:19:28.583427 3151 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:28.583509 3151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:28.583654 3151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:28.585163 3151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "83777bb9f6fa4b93967dc618e31c4fd9"
format_stamp: "Formatted at 2026-09-24 00:19:28 on dist-test-slave-777x"
server_key: "bce7ab565209c330c8b1355b56555896"
server_key_iv: "1e508af3de79145f6c88a82a4f4b8e07"
server_key_version: "encryptionkey@0"
I20260924 00:19:28.585597 3151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "83777bb9f6fa4b93967dc618e31c4fd9"
format_stamp: "Formatted at 2026-09-24 00:19:28 on dist-test-slave-777x"
server_key: "bce7ab565209c330c8b1355b56555896"
server_key_iv: "1e508af3de79145f6c88a82a4f4b8e07"
server_key_version: "encryptionkey@0"
I20260924 00:19:28.586944 3151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:19:28.587832 3166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:28.588035 3151 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:28.588135 3151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "83777bb9f6fa4b93967dc618e31c4fd9"
format_stamp: "Formatted at 2026-09-24 00:19:28 on dist-test-slave-777x"
server_key: "bce7ab565209c330c8b1355b56555896"
server_key_iv: "1e508af3de79145f6c88a82a4f4b8e07"
server_key_version: "encryptionkey@0"
I20260924 00:19:28.588229 3151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:28.599440 3151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:28.601560 3151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:28.601677 3151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:28.601891 3151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:28.602229 3151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:28.602269 3151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:28.602291 3151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:28.602304 3151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:28.608075 3151 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45081
I20260924 00:19:28.608155 3279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45081 every 8 connection(s)
I20260924 00:19:28.608613 3151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:19:28.615182 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 3151
I20260924 00:19:28.615294 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:19:28.615504 392 external_mini_cluster.cc:1428] Setting key 96cd817c7823e91ae29b1f717c7f72bc
Sep 24 00:19:28 dist-test-slave-777x krb5kdc[2776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:28.619426 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:28.609447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44867 (local address 127.0.98.62:40497)
0924 00:19:28.609525 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:28.609527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:28.609761 (+ 234us) server_negotiation.cc:407] Connection header received
0924 00:19:28.610696 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:28.610697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:28.610709 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:28.610786 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:28.611423 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:28.611963 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:28.613415 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:28.613628 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:28.613696 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:28.616541 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:28.616542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:28.616543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:28.616551 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:28.618036 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:28.618506 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:28.618507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:28.618507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:28.618578 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:28.618833 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:28.618834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:28.618834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:28.618947 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:28.618997 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:28.619271 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 00:19:28.619342 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 00:19:28.619585 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:28.609433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40497 (local address 127.0.98.3:44867)
0924 00:19:28.609598 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:28.609641 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:28.610568 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:28.610846 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:28.610850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:28.611126 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:28.611330 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:28.611334 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:28.612222 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:28.612224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:28.613296 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:28.613298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:28.613753 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:28.613755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:28.613886 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:28.614451 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:28.614460 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:28.616450 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:28.618140 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:28.618147 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:28.618150 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:28.618414 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:28.618655 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:28.618656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:28.618656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:28.618780 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:28.618993 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:28.618996 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:28.619132 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 00:19:28.619244 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 00:19:28.619381 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":100,"threads_started":1}
I20260924 00:19:28.620154 3280 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40497
I20260924 00:19:28.620289 3280 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:28.620471 3280 heartbeater.cc:511] Master 127.0.98.62:40497 requested a full tablet report, sending...
I20260924 00:19:28.620880 2821 ts_manager.cc:194] Registered new tserver with Master: 83777bb9f6fa4b93967dc618e31c4fd9 (127.0.98.3:45081)
I20260924 00:19:28.621263 2821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44867
I20260924 00:19:28.628095 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 00:19:28.639973 2861 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:28.639027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44536 (local address 127.0.98.62:40497)
0924 00:19:28.639095 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:28.639101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:28.639227 (+ 126us) server_negotiation.cc:407] Connection header received
0924 00:19:28.639262 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:28.639263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:28.639274 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:28.639340 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:28.639911 (+ 571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44536: BlockingRecv error: recv got EOF from 127.0.0.1:44536 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 00:19:28.640928 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 2883
I20260924 00:19:28.645426 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 3017
I20260924 00:19:28.649607 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 3151
I20260924 00:19:28.654105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 2789
2026-09-24T00:19:28Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1923 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:19:28 dist-test-slave-777x krb5kdc[3292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:19:28 dist-test-slave-777x krb5kdc[3292](info): set up 2 sockets
Sep 24 00:19:28 dist-test-slave-777x krb5kdc[3292](info): commencing operation
krb5kdc: starting...
W20260924 00:19:29.732493 392 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:19:29 dist-test-slave-777x krb5kdc[3292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:19:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:19:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:19:29.858651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41209
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:29.939397 3305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:29.939621 3305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:29.939644 3305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:29.940971 3305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:19:29.941020 3305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:29.941035 3305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:29.941047 3305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:19:29.941066 3305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:19:29.942618 3305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:29.942876 3305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:29.943132 3305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:29.948163 3310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:29.948253 3313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:29.948359 3305 server_base.cc:1034] running on GCE node
W20260924 00:19:29.948266 3311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:29.948769 3305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:29.949100 3305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:29.950289 3305 hybrid_clock.cc:648] HybridClock initialized: now 1790209169950273 us; error 46 us; skew 500 ppm
Sep 24 00:19:29 dist-test-slave-777x krb5kdc[3292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:29.957218 3305 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:19:29.957738 3305 webserver.cc:460] Webserver started at http://127.0.98.62:33985/ using document root <none> and password file <none>
I20260924 00:19:29.958016 3305 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:29.958096 3305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:29.958266 3305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:29.959462 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "f65d3db967a44d5da1dd1cdc1e788ccd"
format_stamp: "Formatted at 2026-09-24 00:19:29 on dist-test-slave-777x"
server_key: "72795c9c4cdb0429811414b7d037cb89"
server_key_iv: "26b5d2d6f6dd5702079d9b3e3b370267"
server_key_version: "encryptionkey@0"
I20260924 00:19:29.959873 3305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "f65d3db967a44d5da1dd1cdc1e788ccd"
format_stamp: "Formatted at 2026-09-24 00:19:29 on dist-test-slave-777x"
server_key: "72795c9c4cdb0429811414b7d037cb89"
server_key_iv: "26b5d2d6f6dd5702079d9b3e3b370267"
server_key_version: "encryptionkey@0"
I20260924 00:19:29.961414 3305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:19:29.962191 3320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:29.962364 3305 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:19:29.962499 3305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "f65d3db967a44d5da1dd1cdc1e788ccd"
format_stamp: "Formatted at 2026-09-24 00:19:29 on dist-test-slave-777x"
server_key: "72795c9c4cdb0429811414b7d037cb89"
server_key_iv: "26b5d2d6f6dd5702079d9b3e3b370267"
server_key_version: "encryptionkey@0"
I20260924 00:19:29.962587 3305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:29.987023 3305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:29.989430 3305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:29.989635 3305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:29.994225 3305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41209
I20260924 00:19:29.994270 3372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41209 every 8 connection(s)
I20260924 00:19:29.994671 3305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:19:29.995632 3373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:29.997988 3373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd: Bootstrap starting.
I20260924 00:19:29.998549 3373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:29.998788 3373 log.cc:824] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:29.999526 3373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd: No bootstrap required, opened a new log
I20260924 00:19:30.001255 3373 raft_consensus.cc:348] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65d3db967a44d5da1dd1cdc1e788ccd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41209 } }
I20260924 00:19:30.001392 3373 raft_consensus.cc:374] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:30.001415 3373 raft_consensus.cc:729] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65d3db967a44d5da1dd1cdc1e788ccd, State: Initialized, Role: FOLLOWER
I20260924 00:19:30.001559 3373 consensus_queue.cc:260] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65d3db967a44d5da1dd1cdc1e788ccd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41209 } }
I20260924 00:19:30.001637 3373 raft_consensus.cc:388] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:19:30.001660 3373 raft_consensus.cc:482] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:19:30.001688 3373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:30.002435 3373 raft_consensus.cc:504] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65d3db967a44d5da1dd1cdc1e788ccd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41209 } }
I20260924 00:19:30.002521 3373 leader_election.cc:302] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f65d3db967a44d5da1dd1cdc1e788ccd; no voters:
I20260924 00:19:30.002673 3373 leader_election.cc:288] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:19:30.002756 3376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:30.002909 3376 raft_consensus.cc:686] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd [term 1 LEADER]: Becoming Leader. State: Replica: f65d3db967a44d5da1dd1cdc1e788ccd, State: Running, Role: LEADER
I20260924 00:19:30.002986 3373 sys_catalog.cc:564] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:19:30.003105 3376 consensus_queue.cc:237] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65d3db967a44d5da1dd1cdc1e788ccd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41209 } }
I20260924 00:19:30.003517 3377 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f65d3db967a44d5da1dd1cdc1e788ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65d3db967a44d5da1dd1cdc1e788ccd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41209 } } }
I20260924 00:19:30.003517 3378 sys_catalog.cc:455] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f65d3db967a44d5da1dd1cdc1e788ccd. Latest consensus state: current_term: 1 leader_uuid: "f65d3db967a44d5da1dd1cdc1e788ccd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65d3db967a44d5da1dd1cdc1e788ccd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41209 } } }
I20260924 00:19:30.003629 3377 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:30.003887 3385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:19:30.003890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 3305
I20260924 00:19:30.003995 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:19:30.004213 392 external_mini_cluster.cc:1428] Setting key 585376b666f12e03ab3e3e9dfa1de1a3
I20260924 00:19:30.004521 3385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:19:30.003638 3378 sys_catalog.cc:458] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:30.006067 3385 catalog_manager.cc:1347] Generated new cluster ID: a780aad0d5fc4b04b44e7fdf3e700c19
I20260924 00:19:30.006121 3385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:19:30 dist-test-slave-777x krb5kdc[3292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:30.017283 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:30.004888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49914 (local address 127.0.98.62:41209)
0924 00:19:30.005206 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:30.005215 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:30.005251 (+ 36us) server_negotiation.cc:407] Connection header received
0924 00:19:30.005367 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:30.005374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:30.005438 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:30.005591 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:30.008740 (+ 3149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:30.009788 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:30.010564 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:30.011300 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:30.011427 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:30.013900 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:30.013904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:30.013905 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:30.013918 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:30.015649 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:30.016035 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:30.016040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:30.016040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:30.016133 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:30.016383 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:30.016384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:30.016385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:30.016645 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:30.016834 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:30.016902 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 00:19:30.017050 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":85,"threads_started":1}
I20260924 00:19:30.018303 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:19:30.020172 3385 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:19:30.020678 3385 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:19:30.026437 3385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f65d3db967a44d5da1dd1cdc1e788ccd: Generated new TSK 0
I20260924 00:19:30.026628 3385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:19:30.035295 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:30.030604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49918 (local address 127.0.98.62:41209)
0924 00:19:30.030648 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:30.030651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:30.030920 (+ 269us) server_negotiation.cc:407] Connection header received
0924 00:19:30.030995 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:30.030996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:30.031006 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:30.031059 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:30.031602 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:30.032140 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:30.033448 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:30.033694 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:30.033771 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:30.034667 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:30.034669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:30.034670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:19:30.034971 (+ 301us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:30.034994 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:19:30.035003 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:30.035064 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:30.035167 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 00:19:30.035198 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:19:30.037997 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 3305
2026-09-24T00:19:30Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1381 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:19:30 dist-test-slave-777x krb5kdc[3406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:19:30 dist-test-slave-777x krb5kdc[3406](info): set up 2 sockets
Sep 24 00:19:30 dist-test-slave-777x krb5kdc[3406](info): commencing operation
krb5kdc: starting...
W20260924 00:19:31.136689 392 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:19:31 dist-test-slave-777x krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:19:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:19:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:19:31.259312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42919
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:31.343712 3419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:31.344043 3419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:31.344089 3419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:31.345432 3419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:19:31.345508 3419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:31.345537 3419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:31.345572 3419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:19:31.345602 3419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:19:31.347225 3419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:31.347529 3419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:31.347780 3419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:31.353397 3424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:31.353473 3427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:31.353502 3425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:31.353682 3419 server_base.cc:1034] running on GCE node
I20260924 00:19:31.353961 3419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:31.354282 3419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:31.355468 3419 hybrid_clock.cc:648] HybridClock initialized: now 1790209171355457 us; error 40 us; skew 500 ppm
Sep 24 00:19:31 dist-test-slave-777x krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:31.358436 3419 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:19:31.358965 3419 webserver.cc:460] Webserver started at http://127.0.98.62:34399/ using document root <none> and password file <none>
I20260924 00:19:31.359226 3419 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:31.359298 3419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:31.359447 3419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:31.360666 3419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "0ce4389ff9744d139a7d0349d283c392"
format_stamp: "Formatted at 2026-09-24 00:19:31 on dist-test-slave-777x"
server_key: "8ef4d6dd549ccd126cc20fa61326eea9"
server_key_iv: "955a28dbae142cb7edafe55802e46239"
server_key_version: "encryptionkey@0"
I20260924 00:19:31.361052 3419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "0ce4389ff9744d139a7d0349d283c392"
format_stamp: "Formatted at 2026-09-24 00:19:31 on dist-test-slave-777x"
server_key: "8ef4d6dd549ccd126cc20fa61326eea9"
server_key_iv: "955a28dbae142cb7edafe55802e46239"
server_key_version: "encryptionkey@0"
I20260924 00:19:31.362457 3419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:19:31.363312 3434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:31.363484 3419 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:31.363631 3419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "0ce4389ff9744d139a7d0349d283c392"
format_stamp: "Formatted at 2026-09-24 00:19:31 on dist-test-slave-777x"
server_key: "8ef4d6dd549ccd126cc20fa61326eea9"
server_key_iv: "955a28dbae142cb7edafe55802e46239"
server_key_version: "encryptionkey@0"
I20260924 00:19:31.363714 3419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:31.384490 3419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:31.386673 3419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:31.386857 3419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:31.391080 3419 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42919
I20260924 00:19:31.391155 3486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42919 every 8 connection(s)
I20260924 00:19:31.391561 3419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:19:31.392818 3487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:31.394683 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 3419
I20260924 00:19:31.394794 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:19:31.394969 392 external_mini_cluster.cc:1428] Setting key a4defcf77eb6e73846e8258c390cc483
I20260924 00:19:31.396011 3487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392: Bootstrap starting.
I20260924 00:19:31.396690 3487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:31.397091 3487 log.cc:824] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:31.397954 3487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392: No bootstrap required, opened a new log
I20260924 00:19:31.399964 3487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce4389ff9744d139a7d0349d283c392" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42919 } }
I20260924 00:19:31.400115 3487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:31.400141 3487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ce4389ff9744d139a7d0349d283c392, State: Initialized, Role: FOLLOWER
I20260924 00:19:31.400377 3487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce4389ff9744d139a7d0349d283c392" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42919 } }
I20260924 00:19:31.400485 3487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:19:31.400535 3487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:19:31.400592 3487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:31.401198 3487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce4389ff9744d139a7d0349d283c392" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42919 } }
I20260924 00:19:31.401331 3487 leader_election.cc:302] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ce4389ff9744d139a7d0349d283c392; no voters:
I20260924 00:19:31.401546 3487 leader_election.cc:288] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:19:31.401579 3492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:31.401759 3492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392 [term 1 LEADER]: Becoming Leader. State: Replica: 0ce4389ff9744d139a7d0349d283c392, State: Running, Role: LEADER
I20260924 00:19:31.401855 3487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:19:31.401881 3492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce4389ff9744d139a7d0349d283c392" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42919 } }
Sep 24 00:19:31 dist-test-slave-777x krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:31.402411 3493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ce4389ff9744d139a7d0349d283c392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce4389ff9744d139a7d0349d283c392" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42919 } } }
I20260924 00:19:31.402490 3493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:31.402730 3494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ce4389ff9744d139a7d0349d283c392. Latest consensus state: current_term: 1 leader_uuid: "0ce4389ff9744d139a7d0349d283c392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ce4389ff9744d139a7d0349d283c392" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42919 } } }
I20260924 00:19:31.402856 3494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:31.403375 3499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:19:31.404155 3499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:19:31.406296 3499 catalog_manager.cc:1347] Generated new cluster ID: 3ac16ba8e7af4bcab0ff07f0961e5d2f
I20260924 00:19:31.406366 3499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:19:31.407150 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:31.395561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35118 (local address 127.0.98.62:42919)
0924 00:19:31.395820 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:31.395830 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:31.395868 (+ 38us) server_negotiation.cc:407] Connection header received
0924 00:19:31.396437 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:31.396445 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:31.396504 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:31.396647 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:31.397535 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:31.398507 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:31.399285 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:31.399709 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:31.399828 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:31.402713 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:31.402718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:31.402718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:31.402731 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:31.405003 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:31.405450 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:31.405453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:31.405454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:31.405540 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:31.405804 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:31.405805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:31.405805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:31.406134 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:31.406249 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:31.406781 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 00:19:31.406933 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":154,"threads_started":1}
I20260924 00:19:31.422307 3499 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:19:31.422896 3499 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:19:31.431116 3499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ce4389ff9744d139a7d0349d283c392: Generated new TSK 0
I20260924 00:19:31.431360 3499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:19:31.463435 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42919
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:31.557490 3513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:31.557744 3513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:31.557785 3513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:31.559019 3513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:31.559090 3513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:31.559134 3513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:19:31.560717 3513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:31.561007 3513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:31.561244 3513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:31.566672 3521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:31.566682 3518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:31.566848 3513 server_base.cc:1034] running on GCE node
W20260924 00:19:31.566810 3519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:31.567371 3513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:31.567624 3513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:31.568781 3513 hybrid_clock.cc:648] HybridClock initialized: now 1790209171568770 us; error 33 us; skew 500 ppm
Sep 24 00:19:31 dist-test-slave-777x krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:31.571593 3513 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:19:31.572086 3513 webserver.cc:460] Webserver started at http://127.0.98.1:36769/ using document root <none> and password file <none>
I20260924 00:19:31.572368 3513 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:31.572419 3513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:31.572604 3513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:31.573769 3513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "d31713992ca445adad9ff32368499518"
format_stamp: "Formatted at 2026-09-24 00:19:31 on dist-test-slave-777x"
server_key: "9d5ed2413b74093d33dee482938d4330"
server_key_iv: "d53dac6c577eaac7fb425f7617674bc0"
server_key_version: "encryptionkey@0"
I20260924 00:19:31.574182 3513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d31713992ca445adad9ff32368499518"
format_stamp: "Formatted at 2026-09-24 00:19:31 on dist-test-slave-777x"
server_key: "9d5ed2413b74093d33dee482938d4330"
server_key_iv: "d53dac6c577eaac7fb425f7617674bc0"
server_key_version: "encryptionkey@0"
I20260924 00:19:31.575816 3513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:19:31.576794 3528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:31.576994 3513 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:19:31.577103 3513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "d31713992ca445adad9ff32368499518"
format_stamp: "Formatted at 2026-09-24 00:19:31 on dist-test-slave-777x"
server_key: "9d5ed2413b74093d33dee482938d4330"
server_key_iv: "d53dac6c577eaac7fb425f7617674bc0"
server_key_version: "encryptionkey@0"
I20260924 00:19:31.577212 3513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:31.604110 3513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:31.606521 3513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:31.606721 3513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:31.607002 3513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:31.607410 3513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:31.607474 3513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:31.607513 3513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:31.607566 3513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:31.614113 3513 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38659
I20260924 00:19:31.614176 3641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38659 every 8 connection(s)
I20260924 00:19:31.614517 3513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:19:31.619591 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 3513
I20260924 00:19:31.619725 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:19:31.619956 392 external_mini_cluster.cc:1428] Setting key b774f86b115e231719f4cea8b9a7691a
Sep 24 00:19:31 dist-test-slave-777x krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:31.625824 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:31.615618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36141 (local address 127.0.98.62:42919)
0924 00:19:31.615720 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:31.615723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:31.615901 (+ 178us) server_negotiation.cc:407] Connection header received
0924 00:19:31.616510 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:31.616511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:31.616524 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:31.616588 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:31.617681 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:31.618325 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:31.619775 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:31.620087 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:31.620203 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:31.622526 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:31.622527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:31.622527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:31.622535 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:31.623924 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:31.624570 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:31.624571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:31.624571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:31.624637 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:31.625040 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:31.625041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:31.625042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:31.625186 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:31.625263 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:31.625673 (+ 410us) server_negotiation.cc:299] Negotiation successful
0924 00:19:31.625754 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 00:19:31.626143 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:31.615277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42919 (local address 127.0.98.1:36141)
0924 00:19:31.615734 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:31.615779 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:31.616386 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:31.616658 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:31.616662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:31.617339 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:31.617562 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:31.617567 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:31.618430 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:31.618431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:31.619363 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:31.619366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:31.620237 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:31.620239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:31.620407 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:31.620982 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:31.620991 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:31.622409 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:31.624046 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:31.624054 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:31.624056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:31.624456 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:31.624717 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:31.624718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:31.624719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:31.624879 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:31.625281 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:31.625288 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:31.625490 (+ 202us) client_negotiation.cc:769] Sending connection context
0924 00:19:31.625635 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 00:19:31.625841 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":74,"threads_started":1}
I20260924 00:19:31.626854 3642 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42919
I20260924 00:19:31.626971 3642 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:31.627179 3642 heartbeater.cc:511] Master 127.0.98.62:42919 requested a full tablet report, sending...
I20260924 00:19:31.627810 3451 ts_manager.cc:194] Registered new tserver with Master: d31713992ca445adad9ff32368499518 (127.0.98.1:38659)
I20260924 00:19:31.628899 3451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36141
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:19:31.647454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42919
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:31.731242 3647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:31.731463 3647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:31.731488 3647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:31.732785 3647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:31.732828 3647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:31.732856 3647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:19:31.734283 3647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:31.734532 3647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:31.734781 3647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:31.739863 3655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:31.739847 3652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:31.739960 3647 server_base.cc:1034] running on GCE node
W20260924 00:19:31.739826 3653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:31.740337 3647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:31.740664 3647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:31.741856 3647 hybrid_clock.cc:648] HybridClock initialized: now 1790209171741841 us; error 45 us; skew 500 ppm
Sep 24 00:19:31 dist-test-slave-777x krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:31.744483 3647 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:19:31.744884 3647 webserver.cc:460] Webserver started at http://127.0.98.2:40061/ using document root <none> and password file <none>
I20260924 00:19:31.745085 3647 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:31.745126 3647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:31.745215 3647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:31.746193 3647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "a73be841778340b692d3e58d43b40f95"
format_stamp: "Formatted at 2026-09-24 00:19:31 on dist-test-slave-777x"
server_key: "a7cb6f54f695f75e41cf9bc09bd17eaf"
server_key_iv: "64455cecaccdd91ed8d43696cd047890"
server_key_version: "encryptionkey@0"
I20260924 00:19:31.746519 3647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a73be841778340b692d3e58d43b40f95"
format_stamp: "Formatted at 2026-09-24 00:19:31 on dist-test-slave-777x"
server_key: "a7cb6f54f695f75e41cf9bc09bd17eaf"
server_key_iv: "64455cecaccdd91ed8d43696cd047890"
server_key_version: "encryptionkey@0"
I20260924 00:19:31.747869 3647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 00:19:31.748744 3662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:31.748941 3647 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:19:31.749027 3647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "a73be841778340b692d3e58d43b40f95"
format_stamp: "Formatted at 2026-09-24 00:19:31 on dist-test-slave-777x"
server_key: "a7cb6f54f695f75e41cf9bc09bd17eaf"
server_key_iv: "64455cecaccdd91ed8d43696cd047890"
server_key_version: "encryptionkey@0"
I20260924 00:19:31.749083 3647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:31.760762 3647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:31.762928 3647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:31.763034 3647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:31.763203 3647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:31.763442 3647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:31.763471 3647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:31.763494 3647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:31.763506 3647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:31.769400 3647 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38175
I20260924 00:19:31.769483 3775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38175 every 8 connection(s)
I20260924 00:19:31.769799 3647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:19:31.772789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 3647
I20260924 00:19:31.772881 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:19:31.773099 392 external_mini_cluster.cc:1428] Setting key 8de1457edcbfdd746be5b1eab1fb5485
Sep 24 00:19:31 dist-test-slave-777x krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:31.781570 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:31.770635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55475 (local address 127.0.98.62:42919)
0924 00:19:31.770684 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:31.770686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:31.770876 (+ 190us) server_negotiation.cc:407] Connection header received
0924 00:19:31.771434 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:31.771435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:31.771448 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:31.771506 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:31.772599 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:31.773288 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:31.774825 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:31.775008 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:31.775112 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:31.777839 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:31.777841 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:31.777841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:31.777851 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:31.779527 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:31.780052 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:31.780053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:31.780054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:31.780125 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:31.780595 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:31.780596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:31.780596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:31.780757 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:31.780838 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:31.781347 (+ 509us) server_negotiation.cc:299] Negotiation successful
0924 00:19:31.781435 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:19:31.781808 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:31.770573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42919 (local address 127.0.98.2:55475)
0924 00:19:31.770752 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:31.770787 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:31.771322 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:31.771595 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:31.771598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:31.772283 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:31.772486 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:31.772491 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:31.773416 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:31.773417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:31.774734 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:31.774737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:31.775120 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:31.775121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:31.775305 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:31.775952 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:31.775961 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:31.777718 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:31.779650 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:31.779657 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:31.779660 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:31.779944 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:31.780272 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:31.780273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:31.780274 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:31.780470 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:31.780850 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:31.780856 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:31.781160 (+ 304us) client_negotiation.cc:769] Sending connection context
0924 00:19:31.781268 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 00:19:31.781440 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 00:19:31.782367 3776 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42919
I20260924 00:19:31.782469 3776 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:31.782696 3776 heartbeater.cc:511] Master 127.0.98.62:42919 requested a full tablet report, sending...
I20260924 00:19:31.783145 3451 ts_manager.cc:194] Registered new tserver with Master: a73be841778340b692d3e58d43b40f95 (127.0.98.2:38175)
I20260924 00:19:31.783720 3451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:19:31.804874 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42919
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:31.888921 3781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:31.889154 3781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:31.889180 3781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:31.890410 3781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:31.890472 3781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:31.890507 3781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:19:31.891996 3781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:31.892263 3781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:31.892529 3781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:31.897786 3789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:31.897814 3787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:31.897820 3786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:31.898092 3781 server_base.cc:1034] running on GCE node
I20260924 00:19:31.898336 3781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:31.898629 3781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:31.899781 3781 hybrid_clock.cc:648] HybridClock initialized: now 1790209171899759 us; error 39 us; skew 500 ppm
Sep 24 00:19:31 dist-test-slave-777x krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:31.902635 3781 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:19:31.903123 3781 webserver.cc:460] Webserver started at http://127.0.98.3:38479/ using document root <none> and password file <none>
I20260924 00:19:31.903401 3781 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:31.903460 3781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:31.903779 3781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:31.904917 3781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "1d51eadcc03e4982a55f6199f49b500d"
format_stamp: "Formatted at 2026-09-24 00:19:31 on dist-test-slave-777x"
server_key: "d65f81d920053326c94f177a335eee2d"
server_key_iv: "e065de454d0b534a27f262f54fe06536"
server_key_version: "encryptionkey@0"
I20260924 00:19:31.905280 3781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1d51eadcc03e4982a55f6199f49b500d"
format_stamp: "Formatted at 2026-09-24 00:19:31 on dist-test-slave-777x"
server_key: "d65f81d920053326c94f177a335eee2d"
server_key_iv: "e065de454d0b534a27f262f54fe06536"
server_key_version: "encryptionkey@0"
I20260924 00:19:31.906738 3781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:19:31.907740 3796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:31.908000 3781 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:31.908111 3781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "1d51eadcc03e4982a55f6199f49b500d"
format_stamp: "Formatted at 2026-09-24 00:19:31 on dist-test-slave-777x"
server_key: "d65f81d920053326c94f177a335eee2d"
server_key_iv: "e065de454d0b534a27f262f54fe06536"
server_key_version: "encryptionkey@0"
I20260924 00:19:31.908247 3781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:31.917197 3781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:31.919421 3781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:31.919597 3781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:31.919862 3781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:31.920291 3781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:31.920348 3781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:31.920394 3781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:31.920424 3781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:31.926224 3781 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34121
I20260924 00:19:31.926265 3909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34121 every 8 connection(s)
I20260924 00:19:31.926635 3781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:19:31.930043 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 3781
I20260924 00:19:31.930160 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:19:31.930342 392 external_mini_cluster.cc:1428] Setting key fc75abf30a2f190ce3653d501974c407
Sep 24 00:19:31 dist-test-slave-777x krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:31.936487 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:31.927403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44341 (local address 127.0.98.62:42919)
0924 00:19:31.927448 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:31.927450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:31.927675 (+ 225us) server_negotiation.cc:407] Connection header received
0924 00:19:31.928141 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:31.928142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:31.928154 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:31.928282 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:31.929276 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:31.929808 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:31.931128 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:31.931405 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:31.931459 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:31.933726 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:31.933727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:31.933727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:31.933734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:31.935104 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:31.935534 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:31.935535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:31.935535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:31.935592 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:31.935851 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:31.935852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:31.935852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:31.935955 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:31.936005 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:31.936330 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 00:19:31.936411 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:19:31.936619 3912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:31.927358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42919 (local address 127.0.98.3:44341)
0924 00:19:31.927516 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:31.927559 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:31.928048 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:31.928344 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:31.928348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:31.928927 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:31.929176 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:31.929181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:31.929933 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:31.929934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:31.931040 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:31.931043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:31.931481 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:31.931481 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:31.931615 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:31.932259 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:31.932269 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:31.933640 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:31.935217 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:31.935222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:31.935224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:31.935452 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:31.935687 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:31.935687 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:31.935688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:31.935792 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:31.936011 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:31.936014 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:31.936138 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 00:19:31.936288 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 00:19:31.936413 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
I20260924 00:19:31.937181 3910 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42919
I20260924 00:19:31.937357 3910 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:31.937604 3910 heartbeater.cc:511] Master 127.0.98.62:42919 requested a full tablet report, sending...
I20260924 00:19:31.938079 3451 ts_manager.cc:194] Registered new tserver with Master: 1d51eadcc03e4982a55f6199f49b500d (127.0.98.3:34121)
I20260924 00:19:31.938429 3451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44341
I20260924 00:19:31.942066 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 00:19:31 dist-test-slave-777x krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209171, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 00:19:31 dist-test-slave-777x krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209171, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:31.965657 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:31.957854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35124 (local address 127.0.98.62:42919)
0924 00:19:31.957939 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:31.957941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:31.957954 (+ 13us) server_negotiation.cc:407] Connection header received
0924 00:19:31.958033 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:31.958034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:31.958044 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:31.958096 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:31.958687 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:31.959204 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:31.959998 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:31.960266 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:31.960330 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:31.962887 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:31.962888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:31.962888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:31.962895 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:31.964370 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:31.964702 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:31.964703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:31.964704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:31.964767 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:31.964947 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:31.964948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:31.964948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:31.965169 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:31.965257 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:31.965508 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 00:19:31.965587 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 00:19:31.966015 3451 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35124
I20260924 00:19:31.966526 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 3513
I20260924 00:19:31.971992 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 3647
I20260924 00:19:31.976620 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 3781
I20260924 00:19:31.980903 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 3419
2026-09-24T00:19:31Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1945 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:19:32 dist-test-slave-777x krb5kdc[3922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:19:32 dist-test-slave-777x krb5kdc[3922](info): set up 2 sockets
Sep 24 00:19:32 dist-test-slave-777x krb5kdc[3922](info): commencing operation
krb5kdc: starting...
W20260924 00:19:33.092722 392 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:19:33 dist-test-slave-777x krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:19:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:19:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:19:33.220268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33697
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:33.308709 3935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:33.308974 3935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:33.309012 3935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:33.310288 3935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:19:33.310360 3935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:33.310386 3935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:33.310410 3935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:19:33.310441 3935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:19:33.311995 3935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:33.312320 3935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:33.312564 3935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:33.317550 3943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:33.317755 3935 server_base.cc:1034] running on GCE node
W20260924 00:19:33.317548 3941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:33.317566 3940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:33.318200 3935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:33.318569 3935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:33.319758 3935 hybrid_clock.cc:648] HybridClock initialized: now 1790209173319745 us; error 30 us; skew 500 ppm
Sep 24 00:19:33 dist-test-slave-777x krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:33.322613 3935 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:19:33.323112 3935 webserver.cc:460] Webserver started at http://127.0.98.62:41375/ using document root <none> and password file <none>
I20260924 00:19:33.323359 3935 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:33.323410 3935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:33.323575 3935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:33.324826 3935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "a4a2fa3a87514b7da7abe99deb1c5100"
format_stamp: "Formatted at 2026-09-24 00:19:33 on dist-test-slave-777x"
server_key: "3c59d6562d77d8234dec96280f827a99"
server_key_iv: "807b9d68d52da22f01c8e76312d4caff"
server_key_version: "encryptionkey@0"
I20260924 00:19:33.325227 3935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "a4a2fa3a87514b7da7abe99deb1c5100"
format_stamp: "Formatted at 2026-09-24 00:19:33 on dist-test-slave-777x"
server_key: "3c59d6562d77d8234dec96280f827a99"
server_key_iv: "807b9d68d52da22f01c8e76312d4caff"
server_key_version: "encryptionkey@0"
I20260924 00:19:33.326690 3935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 00:19:33.327631 3950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:33.327857 3935 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:33.327943 3935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "a4a2fa3a87514b7da7abe99deb1c5100"
format_stamp: "Formatted at 2026-09-24 00:19:33 on dist-test-slave-777x"
server_key: "3c59d6562d77d8234dec96280f827a99"
server_key_iv: "807b9d68d52da22f01c8e76312d4caff"
server_key_version: "encryptionkey@0"
I20260924 00:19:33.328001 3935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:33.334859 3935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:33.336942 3935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:33.337052 3935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:33.341320 3935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33697
I20260924 00:19:33.341387 4002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33697 every 8 connection(s)
I20260924 00:19:33.341735 3935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:19:33.342662 4003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:33.345429 4003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100: Bootstrap starting.
I20260924 00:19:33.345443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 3935
I20260924 00:19:33.345564 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:19:33.345775 392 external_mini_cluster.cc:1428] Setting key 1673fc7c075df20967c6bc0225a850b3
I20260924 00:19:33.346289 4003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:33.346706 4003 log.cc:824] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:33.347673 4003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100: No bootstrap required, opened a new log
I20260924 00:19:33.350044 4003 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4a2fa3a87514b7da7abe99deb1c5100" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } }
I20260924 00:19:33.350231 4003 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:33.350263 4003 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a2fa3a87514b7da7abe99deb1c5100, State: Initialized, Role: FOLLOWER
I20260924 00:19:33.350433 4003 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4a2fa3a87514b7da7abe99deb1c5100" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } }
I20260924 00:19:33.350553 4003 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:19:33.350600 4003 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:19:33.350656 4003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:33.351423 4003 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4a2fa3a87514b7da7abe99deb1c5100" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } }
I20260924 00:19:33.351558 4003 leader_election.cc:302] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4a2fa3a87514b7da7abe99deb1c5100; no voters:
I20260924 00:19:33.351763 4003 leader_election.cc:288] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:19:33.351804 4008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:33.351928 4008 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100 [term 1 LEADER]: Becoming Leader. State: Replica: a4a2fa3a87514b7da7abe99deb1c5100, State: Running, Role: LEADER
I20260924 00:19:33.352043 4003 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:19:33.352031 4008 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4a2fa3a87514b7da7abe99deb1c5100" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } }
I20260924 00:19:33.352424 4008 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4a2fa3a87514b7da7abe99deb1c5100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4a2fa3a87514b7da7abe99deb1c5100" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } }
I20260924 00:19:33.352592 4008 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:33.352815 4008 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4a2fa3a87514b7da7abe99deb1c5100. Latest consensus state: current_term: 1 leader_uuid: "a4a2fa3a87514b7da7abe99deb1c5100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4a2fa3a87514b7da7abe99deb1c5100" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33697 } } }
I20260924 00:19:33.352931 4008 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:33.353370 4016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 00:19:33 dist-test-slave-777x krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:33.353962 4016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:19:33.355700 4016 catalog_manager.cc:1347] Generated new cluster ID: fff922c0a3364f1cbab12d402a07e778
I20260924 00:19:33.355756 4016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:19:33.358124 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:33.346374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51428 (local address 127.0.98.62:33697)
0924 00:19:33.346559 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:33.346567 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:33.346594 (+ 27us) server_negotiation.cc:407] Connection header received
0924 00:19:33.347197 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:33.347207 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:33.347276 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:33.347401 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:33.348365 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:33.349293 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:33.350413 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:33.350946 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:33.351030 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:33.354289 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:33.354299 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:33.354299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:33.354311 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:33.356402 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:33.356840 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:33.356843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:33.356844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:33.356941 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:33.357152 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:33.357160 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:33.357161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:33.357415 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:33.357587 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:33.357696 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 00:19:33.357836 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
I20260924 00:19:33.405941 4016 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:19:33.406843 4016 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:19:33.413513 4016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4a2fa3a87514b7da7abe99deb1c5100: Generated new TSK 0
I20260924 00:19:33.413757 4016 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:19:33.451707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33697
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:33.540045 4029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:33.540352 4029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:33.540385 4029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:33.541673 4029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:33.541726 4029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:33.541755 4029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:19:33.543308 4029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:33.543587 4029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:33.543845 4029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:33.549229 4034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:33.549233 4035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:33.549234 4037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:33.549872 4029 server_base.cc:1034] running on GCE node
I20260924 00:19:33.550108 4029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:33.550346 4029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:33.551537 4029 hybrid_clock.cc:648] HybridClock initialized: now 1790209173551516 us; error 36 us; skew 500 ppm
Sep 24 00:19:33 dist-test-slave-777x krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:33.554586 4029 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:19:33.555137 4029 webserver.cc:460] Webserver started at http://127.0.98.1:41547/ using document root <none> and password file <none>
I20260924 00:19:33.555392 4029 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:33.555464 4029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:33.555622 4029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:33.556856 4029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "139801ecb9e04464984976b69c8cfcfb"
format_stamp: "Formatted at 2026-09-24 00:19:33 on dist-test-slave-777x"
server_key: "829a9bb73e3a3a53350130f3884c4997"
server_key_iv: "9d18718a8c3a2487685c379d7639cad8"
server_key_version: "encryptionkey@0"
I20260924 00:19:33.557245 4029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "139801ecb9e04464984976b69c8cfcfb"
format_stamp: "Formatted at 2026-09-24 00:19:33 on dist-test-slave-777x"
server_key: "829a9bb73e3a3a53350130f3884c4997"
server_key_iv: "9d18718a8c3a2487685c379d7639cad8"
server_key_version: "encryptionkey@0"
I20260924 00:19:33.558640 4029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:19:33.559526 4044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:33.559742 4029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:33.559859 4029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "139801ecb9e04464984976b69c8cfcfb"
format_stamp: "Formatted at 2026-09-24 00:19:33 on dist-test-slave-777x"
server_key: "829a9bb73e3a3a53350130f3884c4997"
server_key_iv: "9d18718a8c3a2487685c379d7639cad8"
server_key_version: "encryptionkey@0"
I20260924 00:19:33.559937 4029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:33.576388 4029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:33.579270 4029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:33.579479 4029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:33.579829 4029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:33.580279 4029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:33.580343 4029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:33.580389 4029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:33.580420 4029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:33.587108 4029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34037
I20260924 00:19:33.587154 4157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34037 every 8 connection(s)
I20260924 00:19:33.587528 4029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:19:33.588210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 4029
I20260924 00:19:33.588361 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:19:33.588634 392 external_mini_cluster.cc:1428] Setting key a8b0b19d141010791f2b1ad9a26663bd
Sep 24 00:19:33 dist-test-slave-777x krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 00:19:33.600142 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:33.588437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50549 (local address 127.0.98.62:33697)
0924 00:19:33.588494 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:33.588497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:33.588853 (+ 356us) server_negotiation.cc:407] Connection header received
0924 00:19:33.590273 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:33.590275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:33.590285 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:33.590343 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:33.591391 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:33.592404 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:33.593538 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:33.593962 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:33.594054 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:33.597022 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:33.597023 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:33.597024 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:33.597033 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:33.598663 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:33.599111 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:33.599112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:33.599113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:33.599176 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:33.599450 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:33.599451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:33.599451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:33.599577 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:33.599657 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:33.599932 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 00:19:33.600029 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 00:19:33.600348 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:33.588434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33697 (local address 127.0.98.1:50549)
0924 00:19:33.588668 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:33.588730 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:33.590116 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:33.590439 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:33.590445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:33.590925 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:33.591263 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:33.591273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:33.592523 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:33.592524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:33.593429 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:33.593432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:33.594118 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:33.594119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:33.594239 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:33.594909 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:33.594922 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:33.596902 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:33.598771 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:33.598778 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:33.598781 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:33.599022 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:33.599246 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:33.599247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:33.599247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:33.599377 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:33.599647 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:33.599653 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:33.599797 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 00:19:33.599912 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 00:19:33.600084 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":98,"threads_started":1}
I20260924 00:19:33.601004 4158 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 00:19:33.601203 4158 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:33.601403 4158 heartbeater.cc:511] Master 127.0.98.62:33697 requested a full tablet report, sending...
I20260924 00:19:33.602099 3967 ts_manager.cc:194] Registered new tserver with Master: 139801ecb9e04464984976b69c8cfcfb (127.0.98.1:34037)
I20260924 00:19:33.603080 3967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:19:33.620517 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33697
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:33.702175 4163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:33.702387 4163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:33.702411 4163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:33.703509 4163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:33.703545 4163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:33.703572 4163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:19:33.705000 4163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:33.705195 4163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:33.705367 4163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:33.710543 4168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:33.710634 4163 server_base.cc:1034] running on GCE node
W20260924 00:19:33.710538 4169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:33.710583 4171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:33.711058 4163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:33.711304 4163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:33.712455 4163 hybrid_clock.cc:648] HybridClock initialized: now 1790209173712441 us; error 29 us; skew 500 ppm
Sep 24 00:19:33 dist-test-slave-777x krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:33.715174 4163 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:19:33.715638 4163 webserver.cc:460] Webserver started at http://127.0.98.2:45701/ using document root <none> and password file <none>
I20260924 00:19:33.715828 4163 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:33.715868 4163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:33.715960 4163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:33.717027 4163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "265ff9b53dd34d1aab911114f4b569b4"
format_stamp: "Formatted at 2026-09-24 00:19:33 on dist-test-slave-777x"
server_key: "072490dffef98a941bad42bc87edc2f3"
server_key_iv: "bb4da6b382b7b7a59dd972ad543f684e"
server_key_version: "encryptionkey@0"
I20260924 00:19:33.717319 4163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "265ff9b53dd34d1aab911114f4b569b4"
format_stamp: "Formatted at 2026-09-24 00:19:33 on dist-test-slave-777x"
server_key: "072490dffef98a941bad42bc87edc2f3"
server_key_iv: "bb4da6b382b7b7a59dd972ad543f684e"
server_key_version: "encryptionkey@0"
I20260924 00:19:33.718554 4163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 00:19:33.719501 4178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:33.719825 4163 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:33.719905 4163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "265ff9b53dd34d1aab911114f4b569b4"
format_stamp: "Formatted at 2026-09-24 00:19:33 on dist-test-slave-777x"
server_key: "072490dffef98a941bad42bc87edc2f3"
server_key_iv: "bb4da6b382b7b7a59dd972ad543f684e"
server_key_version: "encryptionkey@0"
I20260924 00:19:33.720016 4163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:33.739131 4163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:33.741479 4163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:33.741604 4163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:33.741891 4163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:33.742249 4163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:33.742295 4163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:33.742317 4163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:33.742329 4163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:33.748400 4163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46529
I20260924 00:19:33.748445 4291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46529 every 8 connection(s)
I20260924 00:19:33.748764 4163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:19:33.755604 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 4163
I20260924 00:19:33.755738 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
Sep 24 00:19:33 dist-test-slave-777x krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:33.756000 392 external_mini_cluster.cc:1428] Setting key 2d0ebaf5d4d3a0be31876896adc7e8d9
I20260924 00:19:33.760071 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:33.749538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39411 (local address 127.0.98.62:33697)
0924 00:19:33.749584 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:33.749586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:33.749777 (+ 191us) server_negotiation.cc:407] Connection header received
0924 00:19:33.750686 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:33.750687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:33.750698 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:33.750759 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:33.751343 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:33.751965 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:33.753420 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:33.753640 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:33.753724 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:33.756589 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:33.756590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:33.756590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:33.756598 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:33.758322 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:33.758883 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:33.758884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:33.758884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:33.758940 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:33.759252 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:33.759253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:33.759253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:33.759419 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:33.759496 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:33.759832 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 00:19:33.759934 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:19:33.760097 4294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:33.749476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33697 (local address 127.0.98.2:39411)
0924 00:19:33.749627 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:33.749670 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:33.750591 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:33.750813 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:33.750816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:33.751068 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:33.751268 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:33.751273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:33.752119 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:33.752121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:33.753326 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:33.753329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:33.753779 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:33.753780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:33.753909 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:33.754433 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:33.754442 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:33.756448 (+ 2006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:33.758410 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:33.758417 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:33.758420 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:33.758792 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:33.759016 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:33.759017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:33.759017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:33.759172 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:33.759506 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:33.759513 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:33.759682 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 00:19:33.759781 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 00:19:33.759914 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
I20260924 00:19:33.760684 4292 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33697
I20260924 00:19:33.760813 4292 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:33.761026 4292 heartbeater.cc:511] Master 127.0.98.62:33697 requested a full tablet report, sending...
I20260924 00:19:33.761430 3967 ts_manager.cc:194] Registered new tserver with Master: 265ff9b53dd34d1aab911114f4b569b4 (127.0.98.2:46529)
I20260924 00:19:33.761807 3967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39411
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:19:33.784247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33697
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:33.869917 4297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:33.870215 4297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:33.870271 4297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:33.871536 4297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:33.871613 4297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:33.871661 4297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:19:33.873423 4297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:33.873718 4297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:33.873970 4297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:19:33.879302 4297 server_base.cc:1034] running on GCE node
W20260924 00:19:33.879134 4303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:33.879133 4302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:33.879169 4305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:33.879684 4297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:33.879982 4297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:33.881178 4297 hybrid_clock.cc:648] HybridClock initialized: now 1790209173881151 us; error 64 us; skew 500 ppm
Sep 24 00:19:33 dist-test-slave-777x krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:33.883992 4297 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:19:33.884619 4297 webserver.cc:460] Webserver started at http://127.0.98.3:41697/ using document root <none> and password file <none>
I20260924 00:19:33.884873 4297 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:33.884943 4297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:33.885083 4297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:33.886190 4297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "c2ea6f37f0c84f81a964b611c64c0246"
format_stamp: "Formatted at 2026-09-24 00:19:33 on dist-test-slave-777x"
server_key: "047e739af6fa9e06a6488eaf041e02d6"
server_key_iv: "3b3594f87d34115dcc753e862c750fcd"
server_key_version: "encryptionkey@0"
I20260924 00:19:33.886575 4297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c2ea6f37f0c84f81a964b611c64c0246"
format_stamp: "Formatted at 2026-09-24 00:19:33 on dist-test-slave-777x"
server_key: "047e739af6fa9e06a6488eaf041e02d6"
server_key_iv: "3b3594f87d34115dcc753e862c750fcd"
server_key_version: "encryptionkey@0"
I20260924 00:19:33.888006 4297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:19:33.888912 4312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:33.889094 4297 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:33.889257 4297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "c2ea6f37f0c84f81a964b611c64c0246"
format_stamp: "Formatted at 2026-09-24 00:19:33 on dist-test-slave-777x"
server_key: "047e739af6fa9e06a6488eaf041e02d6"
server_key_iv: "3b3594f87d34115dcc753e862c750fcd"
server_key_version: "encryptionkey@0"
I20260924 00:19:33.889353 4297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:33.911463 4297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:33.913975 4297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:33.914157 4297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:33.914443 4297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:33.914779 4297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:33.914836 4297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:33.914880 4297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:33.914906 4297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:33.920729 4297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35099
I20260924 00:19:33.920812 4425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35099 every 8 connection(s)
I20260924 00:19:33.921137 4297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 00:19:33 dist-test-slave-777x krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:33.929572 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 4297
I20260924 00:19:33.929708 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:19:33.929937 392 external_mini_cluster.cc:1428] Setting key 2e5459b0dcd0b42c8c62a4852e3428fc
I20260924 00:19:33.932034 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:33.921935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47749 (local address 127.0.98.62:33697)
0924 00:19:33.922006 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:33.922008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:33.922267 (+ 259us) server_negotiation.cc:407] Connection header received
0924 00:19:33.923317 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:33.923318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:33.923328 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:33.923391 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:33.924145 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:33.924733 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:33.925820 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:33.926022 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:33.926082 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:33.928584 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:33.928585 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:33.928585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:33.928592 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:33.930256 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:33.930860 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:33.930862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:33.930862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:33.930931 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:33.931294 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:33.931295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:33.931295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:33.931441 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:33.931525 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:33.931859 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 00:19:33.931940 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 00:19:33.932278 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:33.921912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33697 (local address 127.0.98.3:47749)
0924 00:19:33.922136 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:33.922183 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:33.923200 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:33.923447 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:33.923450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:33.923801 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:33.924039 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:33.924045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:33.924829 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:33.924830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:33.925731 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:33.925733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:33.926200 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:33.926201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:33.926337 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:33.926839 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:33.926848 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:33.928481 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:33.930334 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:33.930342 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:33.930344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:33.930739 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:33.931021 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:33.931022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:33.931022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:33.931178 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:33.931532 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:33.931538 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:33.931710 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 00:19:33.931832 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 00:19:33.932000 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":89,"threads_started":1}
I20260924 00:19:33.932773 4426 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33697
I20260924 00:19:33.932853 4426 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:33.932996 4426 heartbeater.cc:511] Master 127.0.98.62:33697 requested a full tablet report, sending...
I20260924 00:19:33.933423 3967 ts_manager.cc:194] Registered new tserver with Master: c2ea6f37f0c84f81a964b611c64c0246 (127.0.98.3:35099)
I20260924 00:19:33.933785 3967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47749
I20260924 00:19:33.942250 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 00:19:33 dist-test-slave-777x krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 00:19:33 dist-test-slave-777x krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:19:33.966221 4433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:33.957842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60222 (local address 127.0.98.1:34037)
0924 00:19:33.957995 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:33.958001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:33.958013 (+ 12us) server_negotiation.cc:407] Connection header received
0924 00:19:33.958040 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:33.958042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:33.958056 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:33.958148 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:33.958964 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:33.959707 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:33.960575 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:33.960918 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:33.960976 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:33.963339 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:33.963353 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:33.963353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:33.963364 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:33.965184 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:33.965513 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:33.965515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:33.965516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:33.965568 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:33.965746 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:33.965746 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:33.965746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:33.965878 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:33.965990 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:33.966044 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 00:19:33.966110 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
Sep 24 00:19:33 dist-test-slave-777x krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:33.974529 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:33.967610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.0.98.62:33697)
0924 00:19:33.967690 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:33.967692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:33.967701 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:19:33.967725 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:33.967726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:33.967743 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:33.967807 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:33.968355 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:33.968849 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:33.969558 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:33.969757 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:33.969837 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:33.972122 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:33.972124 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:33.972124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:33.972131 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:33.973527 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:33.973855 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:33.973856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:33.973856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:33.973913 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:33.974080 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:33.974080 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:33.974080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:33.974237 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:33.974314 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:33.974397 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 00:19:33.974463 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 00:19:33.974818 3967 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51444
I20260924 00:19:33.975307 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 4029
I20260924 00:19:33.980759 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 4163
I20260924 00:19:33.985510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 4297
I20260924 00:19:33.990501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 3935
2026-09-24T00:19:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2008 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:19:34 dist-test-slave-777x krb5kdc[4438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:19:34 dist-test-slave-777x krb5kdc[4438](info): set up 2 sockets
Sep 24 00:19:34 dist-test-slave-777x krb5kdc[4438](info): commencing operation
krb5kdc: starting...
W20260924 00:19:35.076349 392 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:19:35 dist-test-slave-777x krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:19:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:19:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:19:35.208834 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34487
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:35.292659 4451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:35.292928 4451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:35.292969 4451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:35.294296 4451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:19:35.294363 4451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:35.294386 4451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:35.294414 4451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:19:35.294440 4451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:19:35.296161 4451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:35.296487 4451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:35.296725 4451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:35.301939 4456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:35.301994 4451 server_base.cc:1034] running on GCE node
W20260924 00:19:35.302022 4457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:35.301939 4459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:35.302552 4451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:35.302875 4451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:35.304028 4451 hybrid_clock.cc:648] HybridClock initialized: now 1790209175304007 us; error 38 us; skew 500 ppm
Sep 24 00:19:35 dist-test-slave-777x krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:35.306766 4451 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:19:35.307160 4451 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:35.307214 4451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:35.307350 4451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:35.308605 4451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "de7cb863b08f41d1b43a5e788ad7c758"
format_stamp: "Formatted at 2026-09-24 00:19:35 on dist-test-slave-777x"
server_key: "fa145387136e12a64584391ad6dd8602"
server_key_iv: "f557bf59ff6e8ce5f34f5f3ea2763a89"
server_key_version: "encryptionkey@0"
I20260924 00:19:35.309049 4451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "de7cb863b08f41d1b43a5e788ad7c758"
format_stamp: "Formatted at 2026-09-24 00:19:35 on dist-test-slave-777x"
server_key: "fa145387136e12a64584391ad6dd8602"
server_key_iv: "f557bf59ff6e8ce5f34f5f3ea2763a89"
server_key_version: "encryptionkey@0"
I20260924 00:19:35.310523 4451 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:19:35.311492 4465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:35.311787 4451 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:35.311908 4451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "de7cb863b08f41d1b43a5e788ad7c758"
format_stamp: "Formatted at 2026-09-24 00:19:35 on dist-test-slave-777x"
server_key: "fa145387136e12a64584391ad6dd8602"
server_key_iv: "f557bf59ff6e8ce5f34f5f3ea2763a89"
server_key_version: "encryptionkey@0"
I20260924 00:19:35.311995 4451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:35.325534 4451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:35.328568 4451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:35.328759 4451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:35.333444 4451 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34487
I20260924 00:19:35.333463 4517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34487 every 8 connection(s)
I20260924 00:19:35.333841 4451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:19:35.334137 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 4451
I20260924 00:19:35.334270 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:19:35.334460 392 external_mini_cluster.cc:1428] Setting key d03e79ad3944388c6fae1330fcf7ac28
I20260924 00:19:35.335191 4518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:35.338022 4518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758: Bootstrap starting.
I20260924 00:19:35.338712 4518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:35.339026 4518 log.cc:824] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:35.339823 4518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758: No bootstrap required, opened a new log
Sep 24 00:19:35 dist-test-slave-777x krb5kdc[4438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:35.342351 4518 raft_consensus.cc:348] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7cb863b08f41d1b43a5e788ad7c758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34487 } }
I20260924 00:19:35.342514 4518 raft_consensus.cc:374] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:35.342538 4518 raft_consensus.cc:729] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de7cb863b08f41d1b43a5e788ad7c758, State: Initialized, Role: FOLLOWER
I20260924 00:19:35.342711 4518 consensus_queue.cc:260] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7cb863b08f41d1b43a5e788ad7c758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34487 } }
I20260924 00:19:35.342805 4518 raft_consensus.cc:388] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:19:35.342842 4518 raft_consensus.cc:482] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:19:35.342902 4518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:35.343601 4518 raft_consensus.cc:504] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7cb863b08f41d1b43a5e788ad7c758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34487 } }
I20260924 00:19:35.343732 4518 leader_election.cc:302] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de7cb863b08f41d1b43a5e788ad7c758; no voters:
I20260924 00:19:35.343904 4518 leader_election.cc:288] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:19:35.343978 4523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:35.344157 4518 sys_catalog.cc:564] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:19:35.344141 4523 raft_consensus.cc:686] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758 [term 1 LEADER]: Becoming Leader. State: Replica: de7cb863b08f41d1b43a5e788ad7c758, State: Running, Role: LEADER
I20260924 00:19:35.344391 4523 consensus_queue.cc:237] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7cb863b08f41d1b43a5e788ad7c758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34487 } }
I20260924 00:19:35.344846 4524 sys_catalog.cc:455] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de7cb863b08f41d1b43a5e788ad7c758. Latest consensus state: current_term: 1 leader_uuid: "de7cb863b08f41d1b43a5e788ad7c758" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7cb863b08f41d1b43a5e788ad7c758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34487 } } }
I20260924 00:19:35.344967 4524 sys_catalog.cc:458] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:35.345146 4525 sys_catalog.cc:455] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de7cb863b08f41d1b43a5e788ad7c758" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de7cb863b08f41d1b43a5e788ad7c758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34487 } } }
I20260924 00:19:35.345295 4525 sys_catalog.cc:458] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:35.345501 4527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:19:35.346048 4527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:19:35.346516 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:35.335068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33144 (local address 127.0.98.62:34487)
0924 00:19:35.335267 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:35.335278 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:35.335362 (+ 84us) server_negotiation.cc:407] Connection header received
0924 00:19:35.335866 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:35.335872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:35.335919 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:35.336065 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:35.336761 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.337675 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.338549 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.339042 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.339171 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:35.342074 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:35.342079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:35.342079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:35.342092 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:35.344310 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:35.344785 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:35.344789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:35.344789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:35.344901 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:35.345241 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:35.345242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:35.345242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:35.345581 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:35.345745 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:35.346137 (+ 392us) server_negotiation.cc:299] Negotiation successful
0924 00:19:35.346295 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":98,"threads_started":1}
I20260924 00:19:35.348697 4527 catalog_manager.cc:1347] Generated new cluster ID: 741661e029b34d21a010a0d068fb667e
I20260924 00:19:35.348755 4527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:19:35.357102 4527 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:19:35.357998 4527 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:19:35.361445 4527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758: Generated new TSK 0
I20260924 00:19:35.361644 4527 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:19:35.397329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34487
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:35.482470 4544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:35.482702 4544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:35.482728 4544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:35.483843 4544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:35.483880 4544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:35.483916 4544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:19:35.485522 4544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:35.485725 4544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:35.485915 4544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:35.491151 4549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:35.491262 4544 server_base.cc:1034] running on GCE node
W20260924 00:19:35.491210 4552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:35.491120 4550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:35.491590 4544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:35.491804 4544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:35.492965 4544 hybrid_clock.cc:648] HybridClock initialized: now 1790209175492953 us; error 27 us; skew 500 ppm
Sep 24 00:19:35 dist-test-slave-777x krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:35.495814 4544 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:19:35.496300 4544 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:35.496368 4544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:35.496527 4544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:35.497642 4544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "15b737a61a2d45b8b8e72dc6135ef6ff"
format_stamp: "Formatted at 2026-09-24 00:19:35 on dist-test-slave-777x"
server_key: "48452b8206796ffe2633032824c69f1d"
server_key_iv: "ad611b9efd53bd1d5a38ec3cdd2d4e60"
server_key_version: "encryptionkey@0"
I20260924 00:19:35.498021 4544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "15b737a61a2d45b8b8e72dc6135ef6ff"
format_stamp: "Formatted at 2026-09-24 00:19:35 on dist-test-slave-777x"
server_key: "48452b8206796ffe2633032824c69f1d"
server_key_iv: "ad611b9efd53bd1d5a38ec3cdd2d4e60"
server_key_version: "encryptionkey@0"
I20260924 00:19:35.499361 4544 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:19:35.500254 4558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:35.500547 4544 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:35.500653 4544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "15b737a61a2d45b8b8e72dc6135ef6ff"
format_stamp: "Formatted at 2026-09-24 00:19:35 on dist-test-slave-777x"
server_key: "48452b8206796ffe2633032824c69f1d"
server_key_iv: "ad611b9efd53bd1d5a38ec3cdd2d4e60"
server_key_version: "encryptionkey@0"
I20260924 00:19:35.500738 4544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:35.517491 4544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:35.519536 4544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:35.519659 4544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:35.519847 4544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:35.520227 4544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:35.520278 4544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:35.520340 4544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:35.520367 4544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:35.526134 4544 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37675
I20260924 00:19:35.526278 4671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37675 every 8 connection(s)
I20260924 00:19:35.526507 4544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:19:35.532763 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 4544
I20260924 00:19:35.532892 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:19:35.533157 392 external_mini_cluster.cc:1428] Setting key 626f01a82c5345d40c1929020eecb537
Sep 24 00:19:35 dist-test-slave-777x krb5kdc[4438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:35.537873 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:35.527296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34823 (local address 127.0.98.62:34487)
0924 00:19:35.527377 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:35.527380 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:35.527567 (+ 187us) server_negotiation.cc:407] Connection header received
0924 00:19:35.528104 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:35.528105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:35.528116 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:35.528243 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:35.529232 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.529760 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.530801 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.530999 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.531066 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:35.533877 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:35.533879 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:35.533879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:35.533887 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:35.535605 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:35.536467 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:35.536468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:35.536483 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:35.536557 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:35.537077 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:35.537078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:35.537079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:35.537289 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:35.537363 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:35.537684 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 00:19:35.537772 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 00:19:35.538216 4674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:35.527241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34487 (local address 127.0.98.1:34823)
0924 00:19:35.527415 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:35.527458 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:35.527941 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:35.528361 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:35.528364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:35.528623 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:35.529114 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:35.529119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.529851 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.529851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:35.530704 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:35.530706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.531108 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.531109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:35.531315 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:35.531872 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:35.531880 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:35.533722 (+ 1842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:35.535706 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:35.535715 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:35.535718 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:35.536069 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:35.536766 (+ 697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:35.536767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:35.536768 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:35.536964 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:35.537366 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:35.537375 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:35.537538 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 00:19:35.537663 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 00:19:35.537846 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
I20260924 00:19:35.538757 4672 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34487
I20260924 00:19:35.538877 4672 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:35.539069 4672 heartbeater.cc:511] Master 127.0.98.62:34487 requested a full tablet report, sending...
I20260924 00:19:35.539789 4482 ts_manager.cc:194] Registered new tserver with Master: 15b737a61a2d45b8b8e72dc6135ef6ff (127.0.98.1:37675)
I20260924 00:19:35.540568 4482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34823
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:19:35.563546 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34487
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:35.646237 4677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:35.646458 4677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:35.646482 4677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:35.647712 4677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:35.647756 4677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:35.647784 4677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:19:35.649307 4677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:35.649537 4677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:35.649770 4677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:35.655218 4683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:35.655225 4682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:35.655350 4685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:35.655522 4677 server_base.cc:1034] running on GCE node
I20260924 00:19:35.655818 4677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:35.656081 4677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:35.657294 4677 hybrid_clock.cc:648] HybridClock initialized: now 1790209175657219 us; error 42 us; skew 500 ppm
Sep 24 00:19:35 dist-test-slave-777x krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:35.660068 4677 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:19:35.660509 4677 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:35.660568 4677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:35.660693 4677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:35.661861 4677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "8b4a20db70ce4e3e9c17c1fdbf11b7df"
format_stamp: "Formatted at 2026-09-24 00:19:35 on dist-test-slave-777x"
server_key: "a49e94407a5940573c2720313530b8d6"
server_key_iv: "e925be693879e28591352c2356024c4e"
server_key_version: "encryptionkey@0"
I20260924 00:19:35.662216 4677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8b4a20db70ce4e3e9c17c1fdbf11b7df"
format_stamp: "Formatted at 2026-09-24 00:19:35 on dist-test-slave-777x"
server_key: "a49e94407a5940573c2720313530b8d6"
server_key_iv: "e925be693879e28591352c2356024c4e"
server_key_version: "encryptionkey@0"
I20260924 00:19:35.663535 4677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:19:35.664351 4691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:35.664525 4677 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:35.664623 4677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "8b4a20db70ce4e3e9c17c1fdbf11b7df"
format_stamp: "Formatted at 2026-09-24 00:19:35 on dist-test-slave-777x"
server_key: "a49e94407a5940573c2720313530b8d6"
server_key_iv: "e925be693879e28591352c2356024c4e"
server_key_version: "encryptionkey@0"
I20260924 00:19:35.664706 4677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:35.674687 4677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:35.677445 4677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:35.677654 4677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:35.677927 4677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:35.678294 4677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:35.678359 4677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:35.678411 4677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:35.678436 4677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:35.685149 4677 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36397
I20260924 00:19:35.685185 4804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36397 every 8 connection(s)
I20260924 00:19:35.685534 4677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:19:35.688930 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 4677
I20260924 00:19:35.689038 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:19:35.689234 392 external_mini_cluster.cc:1428] Setting key 8eb4be6a50736a7d160d0a1b1f1a92fc
Sep 24 00:19:35 dist-test-slave-777x krb5kdc[4438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:35.696765 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:35.686417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34919 (local address 127.0.98.62:34487)
0924 00:19:35.686528 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:35.686531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:35.686705 (+ 174us) server_negotiation.cc:407] Connection header received
0924 00:19:35.687248 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:35.687249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:35.687269 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:35.687338 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:35.688449 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.689133 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.690565 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.690892 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.690971 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:35.693512 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:35.693514 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:35.693515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:35.693524 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:35.695271 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:35.695686 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:35.695687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:35.695688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:35.695754 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:35.696035 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:35.696036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:35.696036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:35.696220 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:35.696309 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:35.696566 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 00:19:35.696657 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 00:19:35.696877 4807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:35.686322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34487 (local address 127.0.98.2:34919)
0924 00:19:35.686555 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:35.686604 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:35.687138 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:35.687414 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:35.687418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:35.687746 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:35.688351 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:35.688357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.689250 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.689251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:35.690462 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:35.690465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.690982 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.690983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:35.691087 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:35.691795 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:35.691806 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:35.693396 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:35.695375 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:35.695381 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:35.695383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:35.695607 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:35.695846 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:35.695847 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:35.695847 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:35.695959 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:35.696297 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:35.696302 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:35.696426 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 00:19:35.696517 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 00:19:35.696652 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":113,"threads_started":1}
I20260924 00:19:35.697440 4805 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34487
I20260924 00:19:35.697540 4805 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:35.697715 4805 heartbeater.cc:511] Master 127.0.98.62:34487 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 00:19:35.698108 4482 ts_manager.cc:194] Registered new tserver with Master: 8b4a20db70ce4e3e9c17c1fdbf11b7df (127.0.98.2:36397)
I20260924 00:19:35.698514 4482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:19:35.715273 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34487
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:35.800822 4810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:35.801061 4810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:35.801105 4810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:35.802306 4810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:35.802382 4810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:35.802428 4810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:19:35.804072 4810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:35.804368 4810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:35.804651 4810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:35.810037 4815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:35.810027 4816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:35.810027 4818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:35.810155 4810 server_base.cc:1034] running on GCE node
I20260924 00:19:35.810519 4810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:35.810724 4810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:35.811856 4810 hybrid_clock.cc:648] HybridClock initialized: now 1790209175811829 us; error 42 us; skew 500 ppm
Sep 24 00:19:35 dist-test-slave-777x krb5kdc[4438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:35.814476 4810 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:19:35.814877 4810 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:35.814965 4810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:35.815109 4810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:35.816396 4810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "df4644dd46914beaa67620ec2067fd8c"
format_stamp: "Formatted at 2026-09-24 00:19:35 on dist-test-slave-777x"
server_key: "04e9a86c036c41738885d38a1632e0d5"
server_key_iv: "97e89d15fb6ad18728db3869c9d2ef93"
server_key_version: "encryptionkey@0"
I20260924 00:19:35.816761 4810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "df4644dd46914beaa67620ec2067fd8c"
format_stamp: "Formatted at 2026-09-24 00:19:35 on dist-test-slave-777x"
server_key: "04e9a86c036c41738885d38a1632e0d5"
server_key_iv: "97e89d15fb6ad18728db3869c9d2ef93"
server_key_version: "encryptionkey@0"
I20260924 00:19:35.818089 4810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 00:19:35.818917 4824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:35.819118 4810 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:35.819201 4810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "df4644dd46914beaa67620ec2067fd8c"
format_stamp: "Formatted at 2026-09-24 00:19:35 on dist-test-slave-777x"
server_key: "04e9a86c036c41738885d38a1632e0d5"
server_key_iv: "97e89d15fb6ad18728db3869c9d2ef93"
server_key_version: "encryptionkey@0"
I20260924 00:19:35.819267 4810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:35.840765 4810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:35.842839 4810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:35.842958 4810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:35.843152 4810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:35.843427 4810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:35.843462 4810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:35.843495 4810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:35.843508 4810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:35.849858 4810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41721
I20260924 00:19:35.849891 4937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41721 every 8 connection(s)
I20260924 00:19:35.850200 4810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:19:35.854856 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 4810
I20260924 00:19:35.854941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:19:35.855127 392 external_mini_cluster.cc:1428] Setting key 2ec3824629466b59a2aff9a03c18caff
Sep 24 00:19:35 dist-test-slave-777x krb5kdc[4438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209175, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:35.859089 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:35.850972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41589 (local address 127.0.98.62:34487)
0924 00:19:35.851038 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:35.851040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:35.851273 (+ 233us) server_negotiation.cc:407] Connection header received
0924 00:19:35.851742 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:35.851743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:35.851754 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:35.851816 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:35.852768 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.853327 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.854272 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.854453 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.854511 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:35.856771 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:35.856773 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:35.856773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:35.856779 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:35.857960 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:35.858312 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:35.858313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:35.858313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:35.858365 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:35.858576 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:35.858577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:35.858577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:35.858676 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:35.858734 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:35.858964 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 00:19:35.859025 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 00:19:35.859385 4940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:35.850949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34487 (local address 127.0.98.3:41589)
0924 00:19:35.851127 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:35.851172 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:35.851654 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:35.851885 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:35.851888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:35.852144 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:35.852674 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:35.852679 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.853410 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.853411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:35.854193 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:35.854195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.854530 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.854530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:35.854626 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:35.855240 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:35.855250 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:35.856695 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:35.858029 (+ 1334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:35.858035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:35.858037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:35.858241 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:35.858428 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:35.858429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:35.858429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:35.858521 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:35.858736 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:35.858740 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:35.858857 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 00:19:35.858938 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 00:19:35.859056 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":110,"threads_started":1}
I20260924 00:19:35.859824 4938 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34487
I20260924 00:19:35.859941 4938 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:35.860095 4938 heartbeater.cc:511] Master 127.0.98.62:34487 requested a full tablet report, sending...
I20260924 00:19:35.860478 4482 ts_manager.cc:194] Registered new tserver with Master: df4644dd46914beaa67620ec2067fd8c (127.0.98.3:41721)
I20260924 00:19:35.860872 4482 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41589
I20260924 00:19:35.866707 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:19:35.873883 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:35.868055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33148 (local address 127.0.98.62:34487)
0924 00:19:35.868113 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:35.868115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:35.868278 (+ 163us) server_negotiation.cc:407] Connection header received
0924 00:19:35.868322 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:35.868322 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:35.868330 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:35.868383 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:35.868933 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.869429 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.870114 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.870305 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.870357 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:35.871540 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:35.871541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:35.871541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:35.871546 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:35.872783 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:35.873193 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:35.873194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:35.873195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:35.873260 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:35.873442 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:35.873443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:35.873443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:35.873586 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:35.873688 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:35.873746 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 00:19:35.873812 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 00:19:35.875607 4482 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:19:35.877401 4482 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:19:35.883841 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:35.879683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41721 (local address 127.0.0.1:38148)
0924 00:19:35.879879 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:35.879911 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:35.879975 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:35.880324 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:35.880327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:35.880330 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:35.880397 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:35.880399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.881919 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.881920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:35.882826 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:35.882829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.883529 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.883530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:35.883607 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:35.883621 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 00:19:35.883699 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 00:19:35.883752 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":2,"thread_start_us":46,"threads_started":1}
I20260924 00:19:35.883942 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:35.879683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36397 (local address 127.0.0.1:39310)
0924 00:19:35.879867 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:35.879908 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:35.879975 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:35.880430 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:35.880431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:35.880433 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:35.880476 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:35.880477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.881780 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.881781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:35.882892 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:35.882894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.883717 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.883719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:35.883789 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:35.883794 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:19:35.883826 (+ 32us) client_negotiation.cc:240] Negotiation successful
0924 00:19:35.883865 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":26,"thread_start_us":78,"threads_started":1}
I20260924 00:19:35.884069 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:35.879717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37675 (local address 127.0.0.1:46506)
0924 00:19:35.880235 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:35.880244 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:35.880300 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:35.880615 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:35.880615 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:35.880617 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:35.880652 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:35.880653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.881497 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.881498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:35.882764 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:35.882767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.883685 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.883686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:35.883742 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:35.883745 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:19:35.883790 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 00:19:35.883824 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":100,"threads_started":1}
I20260924 00:19:35.884109 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:35.879865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46506 (local address 127.0.98.1:37675)
0924 00:19:35.880108 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:35.880114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:35.880479 (+ 365us) server_negotiation.cc:407] Connection header received
0924 00:19:35.880523 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:35.880525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:35.880541 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:35.880618 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:35.880714 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.881390 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.882861 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.883591 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.883679 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:35.883740 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:35.883823 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 00:19:35.883865 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":82,"threads_started":1}
I20260924 00:19:35.884109 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:35.879938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38148 (local address 127.0.98.3:41721)
0924 00:19:35.880117 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:35.880121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:35.880129 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:19:35.880202 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:35.880204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:35.880222 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:35.880312 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:35.880754 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.881811 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.882897 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.883425 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.883526 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:35.883589 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:35.883972 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 00:19:35.884025 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
I20260924 00:19:35.884687 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:35.880058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.0.98.2:36397)
0924 00:19:35.880258 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:35.880262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:35.880271 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:19:35.880341 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:35.880343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:35.880357 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:35.880462 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:35.880540 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.881680 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.882953 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:35.883624 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:35.884481 (+ 857us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:35.884546 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:35.884571 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 00:19:35.884610 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":86,"threads_started":1}
I20260924 00:19:35.884888 4872 tablet_service.cc:1467] Processing CreateTablet for tablet 8a6ab6819c2d48d78c2e4906500a13b3 (DEFAULT_TABLE table=test-table [id=370017d24c194278aa16b085109da73d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:35.884979 4606 tablet_service.cc:1467] Processing CreateTablet for tablet 8a6ab6819c2d48d78c2e4906500a13b3 (DEFAULT_TABLE table=test-table [id=370017d24c194278aa16b085109da73d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:35.885265 4872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6ab6819c2d48d78c2e4906500a13b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:35.885334 4606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6ab6819c2d48d78c2e4906500a13b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:35.885344 4739 tablet_service.cc:1467] Processing CreateTablet for tablet 8a6ab6819c2d48d78c2e4906500a13b3 (DEFAULT_TABLE table=test-table [id=370017d24c194278aa16b085109da73d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:35.885627 4739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6ab6819c2d48d78c2e4906500a13b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:35.888072 4955 tablet_bootstrap.cc:492] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff: Bootstrap starting.
I20260924 00:19:35.888612 4956 tablet_bootstrap.cc:492] T 8a6ab6819c2d48d78c2e4906500a13b3 P df4644dd46914beaa67620ec2067fd8c: Bootstrap starting.
I20260924 00:19:35.888913 4955 tablet_bootstrap.cc:654] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:35.888909 4957 tablet_bootstrap.cc:492] T 8a6ab6819c2d48d78c2e4906500a13b3 P 8b4a20db70ce4e3e9c17c1fdbf11b7df: Bootstrap starting.
I20260924 00:19:35.889267 4956 tablet_bootstrap.cc:654] T 8a6ab6819c2d48d78c2e4906500a13b3 P df4644dd46914beaa67620ec2067fd8c: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:35.889492 4957 tablet_bootstrap.cc:654] T 8a6ab6819c2d48d78c2e4906500a13b3 P 8b4a20db70ce4e3e9c17c1fdbf11b7df: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:35.889585 4955 log.cc:824] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:35.889865 4956 log.cc:824] T 8a6ab6819c2d48d78c2e4906500a13b3 P df4644dd46914beaa67620ec2067fd8c: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:35.890118 4957 log.cc:824] T 8a6ab6819c2d48d78c2e4906500a13b3 P 8b4a20db70ce4e3e9c17c1fdbf11b7df: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:35.890499 4955 tablet_bootstrap.cc:492] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff: No bootstrap required, opened a new log
I20260924 00:19:35.890576 4955 ts_tablet_manager.cc:1397] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:19:35.890699 4956 tablet_bootstrap.cc:492] T 8a6ab6819c2d48d78c2e4906500a13b3 P df4644dd46914beaa67620ec2067fd8c: No bootstrap required, opened a new log
I20260924 00:19:35.890816 4956 ts_tablet_manager.cc:1397] T 8a6ab6819c2d48d78c2e4906500a13b3 P df4644dd46914beaa67620ec2067fd8c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:19:35.890859 4957 tablet_bootstrap.cc:492] T 8a6ab6819c2d48d78c2e4906500a13b3 P 8b4a20db70ce4e3e9c17c1fdbf11b7df: No bootstrap required, opened a new log
I20260924 00:19:35.890923 4957 ts_tablet_manager.cc:1397] T 8a6ab6819c2d48d78c2e4906500a13b3 P 8b4a20db70ce4e3e9c17c1fdbf11b7df: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:19:35.892316 4955 raft_consensus.cc:348] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4a20db70ce4e3e9c17c1fdbf11b7df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36397 } } peers { permanent_uuid: "15b737a61a2d45b8b8e72dc6135ef6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37675 } } peers { permanent_uuid: "df4644dd46914beaa67620ec2067fd8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41721 } }
I20260924 00:19:35.892469 4955 raft_consensus.cc:374] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:35.892513 4955 raft_consensus.cc:729] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15b737a61a2d45b8b8e72dc6135ef6ff, State: Initialized, Role: FOLLOWER
I20260924 00:19:35.892627 4955 consensus_queue.cc:260] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4a20db70ce4e3e9c17c1fdbf11b7df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36397 } } peers { permanent_uuid: "15b737a61a2d45b8b8e72dc6135ef6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37675 } } peers { permanent_uuid: "df4644dd46914beaa67620ec2067fd8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41721 } }
I20260924 00:19:35.892926 4955 ts_tablet_manager.cc:1428] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 00:19:35.892897 4957 raft_consensus.cc:348] T 8a6ab6819c2d48d78c2e4906500a13b3 P 8b4a20db70ce4e3e9c17c1fdbf11b7df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4a20db70ce4e3e9c17c1fdbf11b7df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36397 } } peers { permanent_uuid: "15b737a61a2d45b8b8e72dc6135ef6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37675 } } peers { permanent_uuid: "df4644dd46914beaa67620ec2067fd8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41721 } }
I20260924 00:19:35.892897 4956 raft_consensus.cc:348] T 8a6ab6819c2d48d78c2e4906500a13b3 P df4644dd46914beaa67620ec2067fd8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4a20db70ce4e3e9c17c1fdbf11b7df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36397 } } peers { permanent_uuid: "15b737a61a2d45b8b8e72dc6135ef6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37675 } } peers { permanent_uuid: "df4644dd46914beaa67620ec2067fd8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41721 } }
I20260924 00:19:35.892935 4672 heartbeater.cc:503] Master 127.0.98.62:34487 was elected leader, sending a full tablet report...
I20260924 00:19:35.893061 4957 raft_consensus.cc:374] T 8a6ab6819c2d48d78c2e4906500a13b3 P 8b4a20db70ce4e3e9c17c1fdbf11b7df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:35.893061 4956 raft_consensus.cc:374] T 8a6ab6819c2d48d78c2e4906500a13b3 P df4644dd46914beaa67620ec2067fd8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:35.893105 4956 raft_consensus.cc:729] T 8a6ab6819c2d48d78c2e4906500a13b3 P df4644dd46914beaa67620ec2067fd8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4644dd46914beaa67620ec2067fd8c, State: Initialized, Role: FOLLOWER
I20260924 00:19:35.893105 4957 raft_consensus.cc:729] T 8a6ab6819c2d48d78c2e4906500a13b3 P 8b4a20db70ce4e3e9c17c1fdbf11b7df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b4a20db70ce4e3e9c17c1fdbf11b7df, State: Initialized, Role: FOLLOWER
I20260924 00:19:35.893222 4957 consensus_queue.cc:260] T 8a6ab6819c2d48d78c2e4906500a13b3 P 8b4a20db70ce4e3e9c17c1fdbf11b7df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4a20db70ce4e3e9c17c1fdbf11b7df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36397 } } peers { permanent_uuid: "15b737a61a2d45b8b8e72dc6135ef6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37675 } } peers { permanent_uuid: "df4644dd46914beaa67620ec2067fd8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41721 } }
I20260924 00:19:35.893222 4956 consensus_queue.cc:260] T 8a6ab6819c2d48d78c2e4906500a13b3 P df4644dd46914beaa67620ec2067fd8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4a20db70ce4e3e9c17c1fdbf11b7df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36397 } } peers { permanent_uuid: "15b737a61a2d45b8b8e72dc6135ef6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37675 } } peers { permanent_uuid: "df4644dd46914beaa67620ec2067fd8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41721 } }
I20260924 00:19:35.893489 4957 ts_tablet_manager.cc:1428] T 8a6ab6819c2d48d78c2e4906500a13b3 P 8b4a20db70ce4e3e9c17c1fdbf11b7df: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:19:35.893489 4956 ts_tablet_manager.cc:1428] T 8a6ab6819c2d48d78c2e4906500a13b3 P df4644dd46914beaa67620ec2067fd8c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:19:35.893605 4938 heartbeater.cc:503] Master 127.0.98.62:34487 was elected leader, sending a full tablet report...
I20260924 00:19:35.893605 4805 heartbeater.cc:503] Master 127.0.98.62:34487 was elected leader, sending a full tablet report...
W20260924 00:19:35.936229 4806 tablet.cc:2369] T 8a6ab6819c2d48d78c2e4906500a13b3 P 8b4a20db70ce4e3e9c17c1fdbf11b7df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:19:36.027277 4673 tablet.cc:2369] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:19:36.066790 4961 raft_consensus.cc:482] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:19:36.066912 4961 raft_consensus.cc:504] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4a20db70ce4e3e9c17c1fdbf11b7df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36397 } } peers { permanent_uuid: "15b737a61a2d45b8b8e72dc6135ef6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37675 } } peers { permanent_uuid: "df4644dd46914beaa67620ec2067fd8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41721 } }
I20260924 00:19:36.067481 4961 leader_election.cc:288] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b4a20db70ce4e3e9c17c1fdbf11b7df (127.0.98.2:36397), df4644dd46914beaa67620ec2067fd8c (127.0.98.3:41721)
I20260924 00:19:36.070542 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.067741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37439 (local address 127.0.98.3:41721)
0924 00:19:36.067816 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:36.067817 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:36.067905 (+ 88us) server_negotiation.cc:407] Connection header received
0924 00:19:36.067988 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:36.067989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:36.068000 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:36.068051 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:36.068278 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:36.068774 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:36.069799 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:36.070185 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:36.070276 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:36.070285 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:36.070437 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 00:19:36.070472 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 00:19:36.070540 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.067824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51635 (local address 127.0.98.2:36397)
0924 00:19:36.067873 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:36.067875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:36.067885 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:19:36.067952 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:36.067953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:36.067965 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:36.068041 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:36.068330 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:36.068808 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:36.069812 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:36.070178 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:36.070265 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:36.070276 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:36.070429 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 00:19:36.070465 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 00:19:36.070568 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.067656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36397 (local address 127.0.98.1:51635)
0924 00:19:36.067825 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:36.067835 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:36.067892 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:36.068045 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:36.068046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:36.068048 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:36.068259 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:36.068262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:36.068900 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:36.068901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:36.069715 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:36.069717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:36.070267 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:36.070268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:36.070339 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:36.070346 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 00:19:36.070423 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 00:19:36.070467 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":62,"thread_start_us":62,"threads_started":1}
I20260924 00:19:36.070565 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.067656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41721 (local address 127.0.98.1:37439)
0924 00:19:36.067818 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:36.067830 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:36.067923 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:36.068051 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:36.068051 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:36.068053 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:36.068193 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:36.068197 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:36.068872 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:36.068873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:36.069704 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:36.069706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:36.070279 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:36.070279 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:36.070355 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:36.070359 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:19:36.070430 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 00:19:36.070473 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":34,"thread_start_us":68,"threads_started":1}
I20260924 00:19:36.070986 4759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6ab6819c2d48d78c2e4906500a13b3" candidate_uuid: "15b737a61a2d45b8b8e72dc6135ef6ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b4a20db70ce4e3e9c17c1fdbf11b7df" is_pre_election: true
I20260924 00:19:36.071179 4759 raft_consensus.cc:2454] T 8a6ab6819c2d48d78c2e4906500a13b3 P 8b4a20db70ce4e3e9c17c1fdbf11b7df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15b737a61a2d45b8b8e72dc6135ef6ff in term 0.
I20260924 00:19:36.071376 4892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6ab6819c2d48d78c2e4906500a13b3" candidate_uuid: "15b737a61a2d45b8b8e72dc6135ef6ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4644dd46914beaa67620ec2067fd8c" is_pre_election: true
I20260924 00:19:36.071419 4561 leader_election.cc:302] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15b737a61a2d45b8b8e72dc6135ef6ff, 8b4a20db70ce4e3e9c17c1fdbf11b7df; no voters:
I20260924 00:19:36.071497 4892 raft_consensus.cc:2454] T 8a6ab6819c2d48d78c2e4906500a13b3 P df4644dd46914beaa67620ec2067fd8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15b737a61a2d45b8b8e72dc6135ef6ff in term 0.
I20260924 00:19:36.071529 4961 raft_consensus.cc:2781] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:19:36.071574 4961 raft_consensus.cc:482] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:19:36.071595 4961 raft_consensus.cc:3037] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:36.072376 4961 raft_consensus.cc:504] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4a20db70ce4e3e9c17c1fdbf11b7df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36397 } } peers { permanent_uuid: "15b737a61a2d45b8b8e72dc6135ef6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37675 } } peers { permanent_uuid: "df4644dd46914beaa67620ec2067fd8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41721 } }
I20260924 00:19:36.072510 4961 leader_election.cc:288] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff [CANDIDATE]: Term 1 election: Requested vote from peers 8b4a20db70ce4e3e9c17c1fdbf11b7df (127.0.98.2:36397), df4644dd46914beaa67620ec2067fd8c (127.0.98.3:41721)
I20260924 00:19:36.072645 4759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6ab6819c2d48d78c2e4906500a13b3" candidate_uuid: "15b737a61a2d45b8b8e72dc6135ef6ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b4a20db70ce4e3e9c17c1fdbf11b7df"
I20260924 00:19:36.072713 4759 raft_consensus.cc:3037] T 8a6ab6819c2d48d78c2e4906500a13b3 P 8b4a20db70ce4e3e9c17c1fdbf11b7df [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:36.072731 4892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6ab6819c2d48d78c2e4906500a13b3" candidate_uuid: "15b737a61a2d45b8b8e72dc6135ef6ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df4644dd46914beaa67620ec2067fd8c"
I20260924 00:19:36.072826 4892 raft_consensus.cc:3037] T 8a6ab6819c2d48d78c2e4906500a13b3 P df4644dd46914beaa67620ec2067fd8c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:36.073451 4759 raft_consensus.cc:2454] T 8a6ab6819c2d48d78c2e4906500a13b3 P 8b4a20db70ce4e3e9c17c1fdbf11b7df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15b737a61a2d45b8b8e72dc6135ef6ff in term 1.
I20260924 00:19:36.073611 4561 leader_election.cc:302] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15b737a61a2d45b8b8e72dc6135ef6ff, 8b4a20db70ce4e3e9c17c1fdbf11b7df; no voters:
I20260924 00:19:36.073614 4892 raft_consensus.cc:2454] T 8a6ab6819c2d48d78c2e4906500a13b3 P df4644dd46914beaa67620ec2067fd8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15b737a61a2d45b8b8e72dc6135ef6ff in term 1.
I20260924 00:19:36.073728 4961 raft_consensus.cc:2781] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:36.073889 4961 raft_consensus.cc:686] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff [term 1 LEADER]: Becoming Leader. State: Replica: 15b737a61a2d45b8b8e72dc6135ef6ff, State: Running, Role: LEADER
I20260924 00:19:36.073999 4961 consensus_queue.cc:237] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4a20db70ce4e3e9c17c1fdbf11b7df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36397 } } peers { permanent_uuid: "15b737a61a2d45b8b8e72dc6135ef6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37675 } } peers { permanent_uuid: "df4644dd46914beaa67620ec2067fd8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41721 } }
I20260924 00:19:36.074883 4481 catalog_manager.cc:5495] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 15b737a61a2d45b8b8e72dc6135ef6ff (127.0.98.1). New cstate: current_term: 1 leader_uuid: "15b737a61a2d45b8b8e72dc6135ef6ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b4a20db70ce4e3e9c17c1fdbf11b7df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15b737a61a2d45b8b8e72dc6135ef6ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df4644dd46914beaa67620ec2067fd8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41721 } health_report { overall_health: UNKNOWN } } }
W20260924 00:19:36.100728 4939 tablet.cc:2369] T 8a6ab6819c2d48d78c2e4906500a13b3 P df4644dd46914beaa67620ec2067fd8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:19:36.115084 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.112512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46520 (local address 127.0.98.1:37675)
0924 00:19:36.112580 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:36.112582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:36.112590 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:19:36.112637 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:36.112637 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:36.112669 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:36.112753 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:36.112897 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:36.113450 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:36.114264 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:36.114462 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:36.114551 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:36.114728 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:36.114841 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:36.114932 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:36.114956 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 00:19:36.114977 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 00:19:36.117051 4759 raft_consensus.cc:1261] T 8a6ab6819c2d48d78c2e4906500a13b3 P 8b4a20db70ce4e3e9c17c1fdbf11b7df [term 1 FOLLOWER]: Refusing update from remote peer 15b737a61a2d45b8b8e72dc6135ef6ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:36.117208 4892 raft_consensus.cc:1261] T 8a6ab6819c2d48d78c2e4906500a13b3 P df4644dd46914beaa67620ec2067fd8c [term 1 FOLLOWER]: Refusing update from remote peer 15b737a61a2d45b8b8e72dc6135ef6ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:36.117480 4966 consensus_queue.cc:1035] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b4a20db70ce4e3e9c17c1fdbf11b7df" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:36.117645 4961 consensus_queue.cc:1035] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "df4644dd46914beaa67620ec2067fd8c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:36.120520 4969 mvcc.cc:205] Tried to move back new op lower bound from 7332696785372639232 to 7332696785199800320. Current Snapshot: MvccSnapshot[applied={T|T < 7332696785372639232}]
I20260924 00:19:36.120636 4970 mvcc.cc:205] Tried to move back new op lower bound from 7332696785372639232 to 7332696785199800320. Current Snapshot: MvccSnapshot[applied={T|T < 7332696785372639232}]
I20260924 00:19:36.121045 4971 mvcc.cc:205] Tried to move back new op lower bound from 7332696785372639232 to 7332696785199800320. Current Snapshot: MvccSnapshot[applied={T|T < 7332696785372639232 or (T in {7332696785372639232})}]
I20260924 00:19:36.123612 4481 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:36.123728 4481 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:36.124568 4481 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de7cb863b08f41d1b43a5e788ad7c758: Sending DeleteTablet for 3 replicas of tablet 8a6ab6819c2d48d78c2e4906500a13b3
I20260924 00:19:36.124969 4872 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a6ab6819c2d48d78c2e4906500a13b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38148
I20260924 00:19:36.124953 4739 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a6ab6819c2d48d78c2e4906500a13b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39310
I20260924 00:19:36.125197 4980 tablet_replica.cc:330] T 8a6ab6819c2d48d78c2e4906500a13b3 P 8b4a20db70ce4e3e9c17c1fdbf11b7df: stopping tablet replica
I20260924 00:19:36.125188 4606 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a6ab6819c2d48d78c2e4906500a13b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46506
I20260924 00:19:36.125334 4980 raft_consensus.cc:2229] T 8a6ab6819c2d48d78c2e4906500a13b3 P 8b4a20db70ce4e3e9c17c1fdbf11b7df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:36.125387 4981 tablet_replica.cc:330] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff: stopping tablet replica
I20260924 00:19:36.125530 4981 raft_consensus.cc:2229] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:19:36.125550 4980 raft_consensus.cc:2258] T 8a6ab6819c2d48d78c2e4906500a13b3 P 8b4a20db70ce4e3e9c17c1fdbf11b7df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:36.125607 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 4544
I20260924 00:19:36.125710 4981 raft_consensus.cc:2258] T 8a6ab6819c2d48d78c2e4906500a13b3 P 15b737a61a2d45b8b8e72dc6135ef6ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:36.126036 4980 ts_tablet_manager.cc:1905] T 8a6ab6819c2d48d78c2e4906500a13b3 P 8b4a20db70ce4e3e9c17c1fdbf11b7df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:36.126276 4979 tablet_replica.cc:330] T 8a6ab6819c2d48d78c2e4906500a13b3 P df4644dd46914beaa67620ec2067fd8c: stopping tablet replica
I20260924 00:19:36.126353 4979 raft_consensus.cc:2229] T 8a6ab6819c2d48d78c2e4906500a13b3 P df4644dd46914beaa67620ec2067fd8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:36.126422 4979 raft_consensus.cc:2258] T 8a6ab6819c2d48d78c2e4906500a13b3 P df4644dd46914beaa67620ec2067fd8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:36.127735 4979 ts_tablet_manager.cc:1905] T 8a6ab6819c2d48d78c2e4906500a13b3 P df4644dd46914beaa67620ec2067fd8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:36.127969 4980 ts_tablet_manager.cc:1918] T 8a6ab6819c2d48d78c2e4906500a13b3 P 8b4a20db70ce4e3e9c17c1fdbf11b7df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:19:36.128096 4980 log.cc:1196] T 8a6ab6819c2d48d78c2e4906500a13b3 P 8b4a20db70ce4e3e9c17c1fdbf11b7df: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-1/wal/wals/8a6ab6819c2d48d78c2e4906500a13b3
I20260924 00:19:36.128535 4980 ts_tablet_manager.cc:1939] T 8a6ab6819c2d48d78c2e4906500a13b3 P 8b4a20db70ce4e3e9c17c1fdbf11b7df: Deleting consensus metadata
I20260924 00:19:36.128954 4468 catalog_manager.cc:4841] TS 8b4a20db70ce4e3e9c17c1fdbf11b7df (127.0.98.2:36397): tablet 8a6ab6819c2d48d78c2e4906500a13b3 (table test-table [id=370017d24c194278aa16b085109da73d]) successfully deleted
I20260924 00:19:36.129849 4979 ts_tablet_manager.cc:1918] T 8a6ab6819c2d48d78c2e4906500a13b3 P df4644dd46914beaa67620ec2067fd8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:19:36.129946 4979 log.cc:1196] T 8a6ab6819c2d48d78c2e4906500a13b3 P df4644dd46914beaa67620ec2067fd8c: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/8a6ab6819c2d48d78c2e4906500a13b3
I20260924 00:19:36.130249 4979 ts_tablet_manager.cc:1939] T 8a6ab6819c2d48d78c2e4906500a13b3 P df4644dd46914beaa67620ec2067fd8c: Deleting consensus metadata
I20260924 00:19:36.130781 4467 catalog_manager.cc:4841] TS df4644dd46914beaa67620ec2067fd8c (127.0.98.3:41721): tablet 8a6ab6819c2d48d78c2e4906500a13b3 (table test-table [id=370017d24c194278aa16b085109da73d]) successfully deleted
W20260924 00:19:36.132267 4466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37675 (error 108)
I20260924 00:19:36.132602 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 4677
I20260924 00:19:36.133267 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.132719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37675 (local address 127.0.0.1:46524)
0924 00:19:36.133187 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:36.133238 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445}
W20260924 00:19:36.133396 4466 catalog_manager.cc:4584] TS 15b737a61a2d45b8b8e72dc6135ef6ff (127.0.98.1:37675): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a6ab6819c2d48d78c2e4906500a13b3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37675: connect: Connection refused (error 111)
I20260924 00:19:36.138353 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 4810
I20260924 00:19:36.143299 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 4451
2026-09-24T00:19:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2153 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T00:19:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:19:36Z Disabled control of system clock
I20260924 00:19:36.170797 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40881
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 00:19:36.251437 4986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:36.251670 4986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:36.251693 4986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:36.252857 4986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:19:36.252898 4986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:36.252911 4986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:36.252924 4986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:19:36.252936 4986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:19:36.254264 4986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40881
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:36.254467 4986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:36.254714 4986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:36.260061 4992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:36.260061 4994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:36.260080 4991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:36.260473 4986 server_base.cc:1034] running on GCE node
I20260924 00:19:36.260726 4986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:36.261013 4986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:36.262202 4986 hybrid_clock.cc:648] HybridClock initialized: now 1790209176262178 us; error 41 us; skew 500 ppm
I20260924 00:19:36.263597 4986 webserver.cc:460] Webserver started at http://127.0.98.62:40899/ using document root <none> and password file <none>
I20260924 00:19:36.263860 4986 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:36.263919 4986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:36.264017 4986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:36.265213 4986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "9cd09f881c964642a07c5c0e9ae24a8b"
format_stamp: "Formatted at 2026-09-24 00:19:36 on dist-test-slave-777x"
server_key: "14090d8ecc9f84fc5f12f1f8f4a004e5"
server_key_iv: "95334c0a84af4ef77608fad87ba1d4c3"
server_key_version: "encryptionkey@0"
I20260924 00:19:36.265545 4986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "9cd09f881c964642a07c5c0e9ae24a8b"
format_stamp: "Formatted at 2026-09-24 00:19:36 on dist-test-slave-777x"
server_key: "14090d8ecc9f84fc5f12f1f8f4a004e5"
server_key_iv: "95334c0a84af4ef77608fad87ba1d4c3"
server_key_version: "encryptionkey@0"
I20260924 00:19:36.267057 4986 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:19:36.267899 5000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:36.268098 4986 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:36.268224 4986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "9cd09f881c964642a07c5c0e9ae24a8b"
format_stamp: "Formatted at 2026-09-24 00:19:36 on dist-test-slave-777x"
server_key: "14090d8ecc9f84fc5f12f1f8f4a004e5"
server_key_iv: "95334c0a84af4ef77608fad87ba1d4c3"
server_key_version: "encryptionkey@0"
I20260924 00:19:36.268333 4986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:36.271732 4986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:36.271978 4986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:36.272079 4986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:36.275763 4986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40881
I20260924 00:19:36.275825 5052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40881 every 8 connection(s)
I20260924 00:19:36.276247 4986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:19:36.277077 5053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:36.280265 5053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b: Bootstrap starting.
I20260924 00:19:36.280968 5053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:36.281298 5053 log.cc:824] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:36.282091 5053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b: No bootstrap required, opened a new log
I20260924 00:19:36.283823 5053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd09f881c964642a07c5c0e9ae24a8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } }
I20260924 00:19:36.283969 5053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:36.283993 5053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd09f881c964642a07c5c0e9ae24a8b, State: Initialized, Role: FOLLOWER
I20260924 00:19:36.284237 5053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd09f881c964642a07c5c0e9ae24a8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } }
I20260924 00:19:36.284314 5053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:19:36.284337 5053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:19:36.284366 5053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:36.284948 5053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd09f881c964642a07c5c0e9ae24a8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } }
I20260924 00:19:36.285036 5053 leader_election.cc:302] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cd09f881c964642a07c5c0e9ae24a8b; no voters:
I20260924 00:19:36.285238 5053 leader_election.cc:288] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:19:36.285296 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 4986
I20260924 00:19:36.285399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:19:36.285399 5056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:36.285526 5053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:19:36.285606 392 external_mini_cluster.cc:1428] Setting key 3e2327a4e6b5aed67538dbd2de8a2ecf
I20260924 00:19:36.285609 5056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b [term 1 LEADER]: Becoming Leader. State: Replica: 9cd09f881c964642a07c5c0e9ae24a8b, State: Running, Role: LEADER
I20260924 00:19:36.285699 5056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd09f881c964642a07c5c0e9ae24a8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } }
I20260924 00:19:36.286442 5057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cd09f881c964642a07c5c0e9ae24a8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd09f881c964642a07c5c0e9ae24a8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } } }
I20260924 00:19:36.286545 5057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:36.286725 5058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cd09f881c964642a07c5c0e9ae24a8b. Latest consensus state: current_term: 1 leader_uuid: "9cd09f881c964642a07c5c0e9ae24a8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cd09f881c964642a07c5c0e9ae24a8b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40881 } } }
I20260924 00:19:36.286798 5058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:36.287629 5068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:19:36.288213 5068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:19:36.288547 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.286314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35666 (local address 127.0.98.62:40881)
0924 00:19:36.286571 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:36.286582 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:36.286622 (+ 40us) server_negotiation.cc:407] Connection header received
0924 00:19:36.286775 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:36.286785 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:36.286934 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:36.287154 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:36.287947 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:36.287951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:36.287951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:19:36.287956 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:36.288000 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:19:36.288014 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:36.288085 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:36.288304 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 00:19:36.288368 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":112,"threads_started":1}
I20260924 00:19:36.289883 5068 catalog_manager.cc:1347] Generated new cluster ID: 5a545e29981340bc95e9975f4be62a31
I20260924 00:19:36.289939 5068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:19:36.305276 5068 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:19:36.305496 5068 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:19:36.311239 5068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b: Generated new TSK 0
I20260924 00:19:36.311512 5068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:19:36.314919 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40881
--builtin_ntp_servers=127.0.98.20:34147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 00:19:36.413954 5077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:36.414201 5077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:36.414239 5077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:36.415453 5077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:36.415522 5077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:36.415568 5077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:19:36.417100 5077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:36.417382 5077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:36.417683 5077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:36.423345 5082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:36.423476 5083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:36.423383 5085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:36.424012 5077 server_base.cc:1034] running on GCE node
I20260924 00:19:36.424252 5077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:36.424501 5077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:36.425670 5077 hybrid_clock.cc:648] HybridClock initialized: now 1790209176425655 us; error 26 us; skew 500 ppm
I20260924 00:19:36.427102 5077 webserver.cc:460] Webserver started at http://127.0.98.1:35829/ using document root <none> and password file <none>
I20260924 00:19:36.427376 5077 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:36.427436 5077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:36.427587 5077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:36.429031 5077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "6824aa31d2ba4f3d9a643376b96be9aa"
format_stamp: "Formatted at 2026-09-24 00:19:36 on dist-test-slave-777x"
server_key: "f5b06d466dd8aad8cbe2a060c21d2498"
server_key_iv: "f50204b130e338937be58579de24baaa"
server_key_version: "encryptionkey@0"
I20260924 00:19:36.429440 5077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6824aa31d2ba4f3d9a643376b96be9aa"
format_stamp: "Formatted at 2026-09-24 00:19:36 on dist-test-slave-777x"
server_key: "f5b06d466dd8aad8cbe2a060c21d2498"
server_key_iv: "f50204b130e338937be58579de24baaa"
server_key_version: "encryptionkey@0"
I20260924 00:19:36.431073 5077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:19:36.432066 5091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:36.432286 5077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:36.432377 5077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "6824aa31d2ba4f3d9a643376b96be9aa"
format_stamp: "Formatted at 2026-09-24 00:19:36 on dist-test-slave-777x"
server_key: "f5b06d466dd8aad8cbe2a060c21d2498"
server_key_iv: "f50204b130e338937be58579de24baaa"
server_key_version: "encryptionkey@0"
I20260924 00:19:36.432452 5077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:36.436911 5077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:36.437193 5077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:36.437312 5077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:36.437495 5077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:36.437786 5077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:36.437824 5077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:36.437848 5077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:36.437866 5077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:36.444766 5077 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40711
I20260924 00:19:36.444803 5204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40711 every 8 connection(s)
I20260924 00:19:36.445145 5077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:19:36.448379 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.445908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59263 (local address 127.0.98.62:40881)
0924 00:19:36.445959 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:36.445961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:36.446179 (+ 218us) server_negotiation.cc:407] Connection header received
0924 00:19:36.446754 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:36.446755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:36.446763 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:36.446822 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:36.448027 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:36.448029 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:36.448029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:19:36.448034 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:36.448051 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:19:36.448056 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:36.448111 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:36.448287 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 00:19:36.448307 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 00:19:36.448589 5207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.445843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40881 (local address 127.0.98.1:59263)
0924 00:19:36.446055 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:36.446087 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:36.446639 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:36.446899 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:36.446903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:36.446910 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:36.447871 (+ 961us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:19:36.447871 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:36.447900 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:19:36.447928 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:36.448108 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:36.448112 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:36.448116 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:19:36.448295 (+ 179us) client_negotiation.cc:240] Negotiation successful
0924 00:19:36.448357 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":67,"threads_started":1}
I20260924 00:19:36.449128 5205 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40881
I20260924 00:19:36.449213 5205 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:36.449365 5205 heartbeater.cc:511] Master 127.0.98.62:40881 requested a full tablet report, sending...
I20260924 00:19:36.449891 5017 ts_manager.cc:194] Registered new tserver with Master: 6824aa31d2ba4f3d9a643376b96be9aa (127.0.98.1:40711)
I20260924 00:19:36.450525 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 5077
I20260924 00:19:36.450647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:19:36.450863 392 external_mini_cluster.cc:1428] Setting key df9a476c47f280f2e1c88a4ae8370eb2
I20260924 00:19:36.451889 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40881
--builtin_ntp_servers=127.0.98.20:34147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 00:19:36.531176 5208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:36.531397 5208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:36.531419 5208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:36.532881 5208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:36.532943 5208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:36.532970 5208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:19:36.534569 5208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:36.534850 5208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:36.535169 5208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:36.540556 5216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:36.540735 5214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:36.540602 5213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:36.540943 5208 server_base.cc:1034] running on GCE node
I20260924 00:19:36.541256 5208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:36.541604 5208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:36.542790 5208 hybrid_clock.cc:648] HybridClock initialized: now 1790209176542757 us; error 49 us; skew 500 ppm
I20260924 00:19:36.544533 5208 webserver.cc:460] Webserver started at http://127.0.98.2:33777/ using document root <none> and password file <none>
I20260924 00:19:36.544821 5208 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:36.544873 5208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:36.545043 5208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:36.546422 5208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "b26f2e09fb9049b5b11c7563e8c44c96"
format_stamp: "Formatted at 2026-09-24 00:19:36 on dist-test-slave-777x"
server_key: "83813a8cdbd4b7888d99f1dd4950a95a"
server_key_iv: "e9d493313f93312a7117579dd80a88d0"
server_key_version: "encryptionkey@0"
I20260924 00:19:36.546876 5208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b26f2e09fb9049b5b11c7563e8c44c96"
format_stamp: "Formatted at 2026-09-24 00:19:36 on dist-test-slave-777x"
server_key: "83813a8cdbd4b7888d99f1dd4950a95a"
server_key_iv: "e9d493313f93312a7117579dd80a88d0"
server_key_version: "encryptionkey@0"
I20260924 00:19:36.548501 5208 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260924 00:19:36.549448 5222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:36.549635 5208 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:36.549746 5208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "b26f2e09fb9049b5b11c7563e8c44c96"
format_stamp: "Formatted at 2026-09-24 00:19:36 on dist-test-slave-777x"
server_key: "83813a8cdbd4b7888d99f1dd4950a95a"
server_key_iv: "e9d493313f93312a7117579dd80a88d0"
server_key_version: "encryptionkey@0"
I20260924 00:19:36.549847 5208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:36.553620 5208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:36.553932 5208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:36.554088 5208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:36.554324 5208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:36.554642 5208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:36.554705 5208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:36.554749 5208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:36.554780 5208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:36.561132 5208 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33971
I20260924 00:19:36.561218 5335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33971 every 8 connection(s)
I20260924 00:19:36.561538 5208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:19:36.565315 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.562414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56867 (local address 127.0.98.62:40881)
0924 00:19:36.562485 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:36.562488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:36.562720 (+ 232us) server_negotiation.cc:407] Connection header received
0924 00:19:36.563377 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:36.563379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:36.563386 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:36.563447 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:36.564985 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:36.564987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:36.564987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:19:36.564993 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:36.565010 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:19:36.565016 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:36.565106 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:36.565216 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 00:19:36.565234 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:19:36.565554 5338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.562367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40881 (local address 127.0.98.2:56867)
0924 00:19:36.562579 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:36.562615 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:36.563231 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:36.563536 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:36.563540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:36.563558 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:36.564808 (+ 1250us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:19:36.564808 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:36.564841 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:19:36.564872 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:36.565105 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:36.565112 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:36.565115 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:19:36.565230 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 00:19:36.565299 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
I20260924 00:19:36.566133 5336 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40881
I20260924 00:19:36.566253 5336 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:36.566414 5336 heartbeater.cc:511] Master 127.0.98.62:40881 requested a full tablet report, sending...
I20260924 00:19:36.566658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 5208
I20260924 00:19:36.566813 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:19:36.566887 5017 ts_manager.cc:194] Registered new tserver with Master: b26f2e09fb9049b5b11c7563e8c44c96 (127.0.98.2:33971)
I20260924 00:19:36.567042 392 external_mini_cluster.cc:1428] Setting key a9ab10a6f1fe9da2a7b3dbf7637a8370
I20260924 00:19:36.568346 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40881
--builtin_ntp_servers=127.0.98.20:34147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 00:19:36.651609 5339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:36.651834 5339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:36.651859 5339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:36.653216 5339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:36.653266 5339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:36.653294 5339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:19:36.654718 5339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:36.654976 5339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:36.655288 5339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:36.660751 5344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:36.660773 5339 server_base.cc:1034] running on GCE node
W20260924 00:19:36.660737 5347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:36.660740 5345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:36.661226 5339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:36.661491 5339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:36.662704 5339 hybrid_clock.cc:648] HybridClock initialized: now 1790209176662679 us; error 44 us; skew 500 ppm
I20260924 00:19:36.664157 5339 webserver.cc:460] Webserver started at http://127.0.98.3:46821/ using document root <none> and password file <none>
I20260924 00:19:36.664505 5339 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:36.664557 5339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:36.664702 5339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:36.665844 5339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "1a9a9ca7297c4ad985dcb23a2b44c3ac"
format_stamp: "Formatted at 2026-09-24 00:19:36 on dist-test-slave-777x"
server_key: "2af78e63a34f653e0279284ea4817197"
server_key_iv: "b80a4f5e14977c0d4ee8adeffc0d91b4"
server_key_version: "encryptionkey@0"
I20260924 00:19:36.666221 5339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1a9a9ca7297c4ad985dcb23a2b44c3ac"
format_stamp: "Formatted at 2026-09-24 00:19:36 on dist-test-slave-777x"
server_key: "2af78e63a34f653e0279284ea4817197"
server_key_iv: "b80a4f5e14977c0d4ee8adeffc0d91b4"
server_key_version: "encryptionkey@0"
I20260924 00:19:36.667623 5339 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:19:36.668497 5353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:36.668673 5339 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:36.668780 5339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "1a9a9ca7297c4ad985dcb23a2b44c3ac"
format_stamp: "Formatted at 2026-09-24 00:19:36 on dist-test-slave-777x"
server_key: "2af78e63a34f653e0279284ea4817197"
server_key_iv: "b80a4f5e14977c0d4ee8adeffc0d91b4"
server_key_version: "encryptionkey@0"
I20260924 00:19:36.668857 5339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:36.672525 5339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:36.672820 5339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:36.672966 5339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:36.673182 5339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:36.673509 5339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:36.673568 5339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:36.673607 5339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:36.673635 5339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:36.679695 5339 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41979
I20260924 00:19:36.679739 5466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41979 every 8 connection(s)
I20260924 00:19:36.680131 5339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:19:36.683250 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.680927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38801 (local address 127.0.98.62:40881)
0924 00:19:36.680977 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:36.680980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:36.681266 (+ 286us) server_negotiation.cc:407] Connection header received
0924 00:19:36.681781 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:36.681782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:36.681788 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:36.681848 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:36.683045 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:36.683047 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:36.683047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:19:36.683051 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:36.683065 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:19:36.683068 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:36.683124 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:36.683183 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 00:19:36.683196 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:19:36.683468 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.680923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40881 (local address 127.0.98.3:38801)
0924 00:19:36.681133 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:36.681168 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:36.681646 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:36.681907 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:36.681910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:36.681918 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:36.682894 (+ 976us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:19:36.682894 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:36.682924 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:19:36.682953 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:36.683124 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:36.683126 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:36.683128 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:19:36.683181 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 00:19:36.683228 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":98,"threads_started":1}
I20260924 00:19:36.683828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 5339
I20260924 00:19:36.683969 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:19:36.684039 5467 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40881
I20260924 00:19:36.684155 5467 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:36.684207 392 external_mini_cluster.cc:1428] Setting key 00dda44989654f14285302648eab5bbd
I20260924 00:19:36.684345 5467 heartbeater.cc:511] Master 127.0.98.62:40881 requested a full tablet report, sending...
I20260924 00:19:36.684756 5017 ts_manager.cc:194] Registered new tserver with Master: 1a9a9ca7297c4ad985dcb23a2b44c3ac (127.0.98.3:41979)
I20260924 00:19:36.685132 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:19:36.687786 5066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.686579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35670 (local address 127.0.98.62:40881)
0924 00:19:36.686625 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:36.686627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:36.686800 (+ 173us) server_negotiation.cc:407] Connection header received
0924 00:19:36.686874 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:36.686875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:36.686881 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:36.686939 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:36.687459 (+ 520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:36.687460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:36.687460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:19:36.687464 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:36.687494 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:19:36.687500 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:36.687565 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:36.687684 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 00:19:36.687700 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 00:19:36.689772 5016 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:19:36.691311 5016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:19:36.695462 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.693976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42532 (local address 127.0.98.1:40711)
0924 00:19:36.694146 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:36.694152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:36.694163 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:19:36.694190 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:36.694193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:36.694202 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:36.694509 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:36.695060 (+ 551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:36.695068 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:36.695068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:19:36.695073 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:36.695102 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:19:36.695134 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:36.695193 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:36.695365 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 00:19:36.695382 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":101,"threads_started":1}
I20260924 00:19:36.695686 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.693612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40711 (local address 127.0.0.1:42532)
0924 00:19:36.693817 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:36.693863 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:36.693917 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:36.694311 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:36.694312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:36.694315 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:36.694964 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:19:36.694965 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:36.694972 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:19:36.694986 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:36.695190 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:36.695196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:36.695199 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:19:36.695332 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 00:19:36.695611 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":77,"threads_started":1}
I20260924 00:19:36.695782 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.693693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39276 (local address 127.0.98.3:41979)
0924 00:19:36.693880 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:36.693884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:36.694022 (+ 138us) server_negotiation.cc:407] Connection header received
0924 00:19:36.694100 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:36.694102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:36.694113 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:36.694189 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:36.694998 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:36.695004 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:36.695004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:19:36.695008 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:36.695042 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:19:36.695063 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:36.695141 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:36.695657 (+ 516us) server_negotiation.cc:299] Negotiation successful
0924 00:19:36.695674 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":118,"threads_started":1}
I20260924 00:19:36.695955 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.693612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41979 (local address 127.0.0.1:39276)
0924 00:19:36.693943 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:36.693953 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:36.694028 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:36.694196 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:36.694202 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:36.694271 (+ 69us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:36.694883 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:19:36.694884 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:36.694898 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:19:36.694916 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:36.695484 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:36.695485 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:36.695486 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 00:19:36.695538 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 00:19:36.695901 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":2,"thread_start_us":76,"threads_started":1}
I20260924 00:19:36.696250 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.693826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33971 (local address 127.0.0.1:43002)
0924 00:19:36.694977 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:36.694988 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:36.695069 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:36.695216 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:36.695217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:36.695220 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:36.695782 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:19:36.695783 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:36.695790 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:19:36.695806 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:36.696061 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:36.696062 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:36.696062 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 00:19:36.696127 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 00:19:36.696140 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1139,"thread_start_us":76,"threads_started":1}
I20260924 00:19:36.696274 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.694263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43002 (local address 127.0.98.2:33971)
0924 00:19:36.694453 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:36.694458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:36.695046 (+ 588us) server_negotiation.cc:407] Connection header received
0924 00:19:36.695126 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:36.695130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:36.695139 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:36.695213 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:36.695898 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:36.695907 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:36.695907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:19:36.695912 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:36.695952 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:19:36.695979 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:36.696035 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:36.696149 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 00:19:36.696196 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":86,"threads_started":1}
I20260924 00:19:36.696507 5139 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc92680a7984bd09d8b942b7b2ad669 (DEFAULT_TABLE table=test-table [id=311f9240f36f4f02bd9a35c48be5b667]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:36.696810 5139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc92680a7984bd09d8b942b7b2ad669. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:36.699252 5481 tablet_bootstrap.cc:492] T 5dc92680a7984bd09d8b942b7b2ad669 P 6824aa31d2ba4f3d9a643376b96be9aa: Bootstrap starting.
I20260924 00:19:36.699838 5401 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc92680a7984bd09d8b942b7b2ad669 (DEFAULT_TABLE table=test-table [id=311f9240f36f4f02bd9a35c48be5b667]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:36.699998 5481 tablet_bootstrap.cc:654] T 5dc92680a7984bd09d8b942b7b2ad669 P 6824aa31d2ba4f3d9a643376b96be9aa: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:36.699918 5270 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc92680a7984bd09d8b942b7b2ad669 (DEFAULT_TABLE table=test-table [id=311f9240f36f4f02bd9a35c48be5b667]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:36.700213 5401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc92680a7984bd09d8b942b7b2ad669. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:36.700278 5270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc92680a7984bd09d8b942b7b2ad669. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:36.700429 5481 log.cc:824] T 5dc92680a7984bd09d8b942b7b2ad669 P 6824aa31d2ba4f3d9a643376b96be9aa: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:36.701277 5481 tablet_bootstrap.cc:492] T 5dc92680a7984bd09d8b942b7b2ad669 P 6824aa31d2ba4f3d9a643376b96be9aa: No bootstrap required, opened a new log
I20260924 00:19:36.701349 5481 ts_tablet_manager.cc:1397] T 5dc92680a7984bd09d8b942b7b2ad669 P 6824aa31d2ba4f3d9a643376b96be9aa: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 00:19:36.703153 5483 tablet_bootstrap.cc:492] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac: Bootstrap starting.
I20260924 00:19:36.703670 5484 tablet_bootstrap.cc:492] T 5dc92680a7984bd09d8b942b7b2ad669 P b26f2e09fb9049b5b11c7563e8c44c96: Bootstrap starting.
I20260924 00:19:36.703816 5483 tablet_bootstrap.cc:654] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:36.704160 5483 log.cc:824] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:36.704088 5481 raft_consensus.cc:348] T 5dc92680a7984bd09d8b942b7b2ad669 P 6824aa31d2ba4f3d9a643376b96be9aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9a9ca7297c4ad985dcb23a2b44c3ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41979 } } peers { permanent_uuid: "b26f2e09fb9049b5b11c7563e8c44c96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33971 } } peers { permanent_uuid: "6824aa31d2ba4f3d9a643376b96be9aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } }
I20260924 00:19:36.704314 5481 raft_consensus.cc:374] T 5dc92680a7984bd09d8b942b7b2ad669 P 6824aa31d2ba4f3d9a643376b96be9aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:36.704360 5481 raft_consensus.cc:729] T 5dc92680a7984bd09d8b942b7b2ad669 P 6824aa31d2ba4f3d9a643376b96be9aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6824aa31d2ba4f3d9a643376b96be9aa, State: Initialized, Role: FOLLOWER
I20260924 00:19:36.704429 5484 tablet_bootstrap.cc:654] T 5dc92680a7984bd09d8b942b7b2ad669 P b26f2e09fb9049b5b11c7563e8c44c96: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:36.704473 5481 consensus_queue.cc:260] T 5dc92680a7984bd09d8b942b7b2ad669 P 6824aa31d2ba4f3d9a643376b96be9aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9a9ca7297c4ad985dcb23a2b44c3ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41979 } } peers { permanent_uuid: "b26f2e09fb9049b5b11c7563e8c44c96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33971 } } peers { permanent_uuid: "6824aa31d2ba4f3d9a643376b96be9aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } }
I20260924 00:19:36.704741 5484 log.cc:824] T 5dc92680a7984bd09d8b942b7b2ad669 P b26f2e09fb9049b5b11c7563e8c44c96: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:36.704782 5205 heartbeater.cc:503] Master 127.0.98.62:40881 was elected leader, sending a full tablet report...
I20260924 00:19:36.704946 5483 tablet_bootstrap.cc:492] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac: No bootstrap required, opened a new log
I20260924 00:19:36.705020 5483 ts_tablet_manager.cc:1397] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:19:36.704780 5481 ts_tablet_manager.cc:1428] T 5dc92680a7984bd09d8b942b7b2ad669 P 6824aa31d2ba4f3d9a643376b96be9aa: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 00:19:36.705657 5484 tablet_bootstrap.cc:492] T 5dc92680a7984bd09d8b942b7b2ad669 P b26f2e09fb9049b5b11c7563e8c44c96: No bootstrap required, opened a new log
I20260924 00:19:36.705734 5484 ts_tablet_manager.cc:1397] T 5dc92680a7984bd09d8b942b7b2ad669 P b26f2e09fb9049b5b11c7563e8c44c96: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:19:36.707690 5483 raft_consensus.cc:348] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9a9ca7297c4ad985dcb23a2b44c3ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41979 } } peers { permanent_uuid: "b26f2e09fb9049b5b11c7563e8c44c96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33971 } } peers { permanent_uuid: "6824aa31d2ba4f3d9a643376b96be9aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } }
I20260924 00:19:36.707907 5483 raft_consensus.cc:374] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:36.707949 5483 raft_consensus.cc:729] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9a9ca7297c4ad985dcb23a2b44c3ac, State: Initialized, Role: FOLLOWER
I20260924 00:19:36.708082 5483 consensus_queue.cc:260] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9a9ca7297c4ad985dcb23a2b44c3ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41979 } } peers { permanent_uuid: "b26f2e09fb9049b5b11c7563e8c44c96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33971 } } peers { permanent_uuid: "6824aa31d2ba4f3d9a643376b96be9aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } }
I20260924 00:19:36.708388 5483 ts_tablet_manager.cc:1428] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:19:36.708420 5467 heartbeater.cc:503] Master 127.0.98.62:40881 was elected leader, sending a full tablet report...
I20260924 00:19:36.708393 5484 raft_consensus.cc:348] T 5dc92680a7984bd09d8b942b7b2ad669 P b26f2e09fb9049b5b11c7563e8c44c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9a9ca7297c4ad985dcb23a2b44c3ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41979 } } peers { permanent_uuid: "b26f2e09fb9049b5b11c7563e8c44c96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33971 } } peers { permanent_uuid: "6824aa31d2ba4f3d9a643376b96be9aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } }
I20260924 00:19:36.708540 5484 raft_consensus.cc:374] T 5dc92680a7984bd09d8b942b7b2ad669 P b26f2e09fb9049b5b11c7563e8c44c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:36.708576 5484 raft_consensus.cc:729] T 5dc92680a7984bd09d8b942b7b2ad669 P b26f2e09fb9049b5b11c7563e8c44c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b26f2e09fb9049b5b11c7563e8c44c96, State: Initialized, Role: FOLLOWER
I20260924 00:19:36.708707 5484 consensus_queue.cc:260] T 5dc92680a7984bd09d8b942b7b2ad669 P b26f2e09fb9049b5b11c7563e8c44c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9a9ca7297c4ad985dcb23a2b44c3ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41979 } } peers { permanent_uuid: "b26f2e09fb9049b5b11c7563e8c44c96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33971 } } peers { permanent_uuid: "6824aa31d2ba4f3d9a643376b96be9aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } }
I20260924 00:19:36.709003 5484 ts_tablet_manager.cc:1428] T 5dc92680a7984bd09d8b942b7b2ad669 P b26f2e09fb9049b5b11c7563e8c44c96: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 00:19:36.709048 5336 heartbeater.cc:503] Master 127.0.98.62:40881 was elected leader, sending a full tablet report...
I20260924 00:19:36.776759 5488 raft_consensus.cc:482] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:19:36.776919 5488 raft_consensus.cc:504] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9a9ca7297c4ad985dcb23a2b44c3ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41979 } } peers { permanent_uuid: "b26f2e09fb9049b5b11c7563e8c44c96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33971 } } peers { permanent_uuid: "6824aa31d2ba4f3d9a643376b96be9aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } }
I20260924 00:19:36.777328 5488 leader_election.cc:288] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b26f2e09fb9049b5b11c7563e8c44c96 (127.0.98.2:33971), 6824aa31d2ba4f3d9a643376b96be9aa (127.0.98.1:40711)
I20260924 00:19:36.778901 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.777527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40711 (local address 127.0.98.3:47809)
0924 00:19:36.777605 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:36.777618 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:36.777663 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:36.777854 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:36.777856 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:36.777859 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:36.778387 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:19:36.778388 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:36.778396 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:19:36.778410 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:36.778706 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:36.778707 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:36.778708 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 00:19:36.778804 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 00:19:36.778819 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 00:19:36.778928 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.777650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47809 (local address 127.0.98.1:40711)
0924 00:19:36.777715 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:36.777717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:36.777734 (+ 17us) server_negotiation.cc:407] Connection header received
0924 00:19:36.777760 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:36.777761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:36.777766 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:36.777850 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:36.778535 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:36.778540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:36.778541 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:19:36.778546 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:36.778567 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:19:36.778572 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:36.778648 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:36.778825 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 00:19:36.778841 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 00:19:36.778995 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.777619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36423 (local address 127.0.98.2:33971)
0924 00:19:36.777668 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:36.777670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:36.777817 (+ 147us) server_negotiation.cc:407] Connection header received
0924 00:19:36.777890 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:36.777891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:36.777898 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:36.777953 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:36.778718 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:36.778720 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:36.778721 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:19:36.778726 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:36.778744 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:19:36.778749 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:36.778812 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:36.778905 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 00:19:36.778919 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:19:36.779039 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.777526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33971 (local address 127.0.98.3:36423)
0924 00:19:36.777725 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:36.777735 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:36.777819 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:36.777956 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:36.777957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:36.777959 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:36.778583 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:19:36.778584 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:36.778590 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:19:36.778604 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:36.778823 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:36.778824 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:36.778824 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 00:19:36.778901 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 00:19:36.778913 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":18,"thread_start_us":127,"threads_started":1}
I20260924 00:19:36.779358 5159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc92680a7984bd09d8b942b7b2ad669" candidate_uuid: "1a9a9ca7297c4ad985dcb23a2b44c3ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6824aa31d2ba4f3d9a643376b96be9aa" is_pre_election: true
I20260924 00:19:36.779546 5159 raft_consensus.cc:2454] T 5dc92680a7984bd09d8b942b7b2ad669 P 6824aa31d2ba4f3d9a643376b96be9aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9a9ca7297c4ad985dcb23a2b44c3ac in term 0.
I20260924 00:19:36.779891 5290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc92680a7984bd09d8b942b7b2ad669" candidate_uuid: "1a9a9ca7297c4ad985dcb23a2b44c3ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b26f2e09fb9049b5b11c7563e8c44c96" is_pre_election: true
I20260924 00:19:36.780014 5290 raft_consensus.cc:2454] T 5dc92680a7984bd09d8b942b7b2ad669 P b26f2e09fb9049b5b11c7563e8c44c96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9a9ca7297c4ad985dcb23a2b44c3ac in term 0.
I20260924 00:19:36.780287 5355 leader_election.cc:302] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9a9ca7297c4ad985dcb23a2b44c3ac, 6824aa31d2ba4f3d9a643376b96be9aa; no voters:
I20260924 00:19:36.780422 5488 raft_consensus.cc:2781] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:19:36.780478 5488 raft_consensus.cc:482] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:19:36.780507 5488 raft_consensus.cc:3037] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:36.781243 5488 raft_consensus.cc:504] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9a9ca7297c4ad985dcb23a2b44c3ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41979 } } peers { permanent_uuid: "b26f2e09fb9049b5b11c7563e8c44c96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33971 } } peers { permanent_uuid: "6824aa31d2ba4f3d9a643376b96be9aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } }
I20260924 00:19:36.781394 5488 leader_election.cc:288] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac [CANDIDATE]: Term 1 election: Requested vote from peers b26f2e09fb9049b5b11c7563e8c44c96 (127.0.98.2:33971), 6824aa31d2ba4f3d9a643376b96be9aa (127.0.98.1:40711)
I20260924 00:19:36.781540 5290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc92680a7984bd09d8b942b7b2ad669" candidate_uuid: "1a9a9ca7297c4ad985dcb23a2b44c3ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b26f2e09fb9049b5b11c7563e8c44c96"
I20260924 00:19:36.781540 5159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc92680a7984bd09d8b942b7b2ad669" candidate_uuid: "1a9a9ca7297c4ad985dcb23a2b44c3ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6824aa31d2ba4f3d9a643376b96be9aa"
I20260924 00:19:36.781620 5290 raft_consensus.cc:3037] T 5dc92680a7984bd09d8b942b7b2ad669 P b26f2e09fb9049b5b11c7563e8c44c96 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:36.781620 5159 raft_consensus.cc:3037] T 5dc92680a7984bd09d8b942b7b2ad669 P 6824aa31d2ba4f3d9a643376b96be9aa [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:36.782518 5290 raft_consensus.cc:2454] T 5dc92680a7984bd09d8b942b7b2ad669 P b26f2e09fb9049b5b11c7563e8c44c96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9a9ca7297c4ad985dcb23a2b44c3ac in term 1.
I20260924 00:19:36.782523 5159 raft_consensus.cc:2454] T 5dc92680a7984bd09d8b942b7b2ad669 P 6824aa31d2ba4f3d9a643376b96be9aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9a9ca7297c4ad985dcb23a2b44c3ac in term 1.
I20260924 00:19:36.782758 5355 leader_election.cc:302] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a9a9ca7297c4ad985dcb23a2b44c3ac, 6824aa31d2ba4f3d9a643376b96be9aa; no voters:
I20260924 00:19:36.782868 5488 raft_consensus.cc:2781] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:36.783067 5488 raft_consensus.cc:686] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac [term 1 LEADER]: Becoming Leader. State: Replica: 1a9a9ca7297c4ad985dcb23a2b44c3ac, State: Running, Role: LEADER
I20260924 00:19:36.783177 5488 consensus_queue.cc:237] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9a9ca7297c4ad985dcb23a2b44c3ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41979 } } peers { permanent_uuid: "b26f2e09fb9049b5b11c7563e8c44c96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33971 } } peers { permanent_uuid: "6824aa31d2ba4f3d9a643376b96be9aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } }
I20260924 00:19:36.784061 5016 catalog_manager.cc:5495] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a9a9ca7297c4ad985dcb23a2b44c3ac (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1a9a9ca7297c4ad985dcb23a2b44c3ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a9a9ca7297c4ad985dcb23a2b44c3ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b26f2e09fb9049b5b11c7563e8c44c96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6824aa31d2ba4f3d9a643376b96be9aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 } health_report { overall_health: UNKNOWN } } }
I20260924 00:19:36.793574 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.792409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39280 (local address 127.0.98.3:41979)
0924 00:19:36.792512 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:36.792515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:36.792524 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:19:36.792605 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:36.792606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:36.792620 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:36.792677 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:36.793397 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:36.793398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:36.793399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:19:36.793403 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:36.793419 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:19:36.793423 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:36.793474 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:36.793504 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 00:19:36.793517 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 00:19:36.795691 5159 raft_consensus.cc:1261] T 5dc92680a7984bd09d8b942b7b2ad669 P 6824aa31d2ba4f3d9a643376b96be9aa [term 1 FOLLOWER]: Refusing update from remote peer 1a9a9ca7297c4ad985dcb23a2b44c3ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:36.795691 5290 raft_consensus.cc:1261] T 5dc92680a7984bd09d8b942b7b2ad669 P b26f2e09fb9049b5b11c7563e8c44c96 [term 1 FOLLOWER]: Refusing update from remote peer 1a9a9ca7297c4ad985dcb23a2b44c3ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:36.796140 5488 consensus_queue.cc:1035] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "6824aa31d2ba4f3d9a643376b96be9aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:36.796422 5491 consensus_queue.cc:1035] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "b26f2e09fb9049b5b11c7563e8c44c96" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:36.799305 5496 mvcc.cc:205] Tried to move back new op lower bound from 7332696788152049664 to 7332696788104609792. Current Snapshot: MvccSnapshot[applied={T|T < 7332696788152049664 or (T in {7332696788152049664})}]
I20260924 00:19:36.799485 5494 mvcc.cc:205] Tried to move back new op lower bound from 7332696788152049664 to 7332696788104609792. Current Snapshot: MvccSnapshot[applied={T|T < 7332696788152049664 or (T in {7332696788152049664})}]
I20260924 00:19:36.800150 5495 mvcc.cc:205] Tried to move back new op lower bound from 7332696788152049664 to 7332696788104609792. Current Snapshot: MvccSnapshot[applied={T|T < 7332696788152049664}]
I20260924 00:19:36.801154 5017 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:36.801258 5017 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:36.802037 5017 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9cd09f881c964642a07c5c0e9ae24a8b: Sending DeleteTablet for 3 replicas of tablet 5dc92680a7984bd09d8b942b7b2ad669
I20260924 00:19:36.802428 5139 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dc92680a7984bd09d8b942b7b2ad669 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:36 UTC) from {username='slave'} at 127.0.0.1:42532
I20260924 00:19:36.802438 5401 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dc92680a7984bd09d8b942b7b2ad669 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:36 UTC) from {username='slave'} at 127.0.0.1:39276
I20260924 00:19:36.802666 5503 tablet_replica.cc:330] T 5dc92680a7984bd09d8b942b7b2ad669 P 6824aa31d2ba4f3d9a643376b96be9aa: stopping tablet replica
I20260924 00:19:36.802667 5270 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dc92680a7984bd09d8b942b7b2ad669 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:36 UTC) from {username='slave'} at 127.0.0.1:43002
I20260924 00:19:36.802773 5503 raft_consensus.cc:2229] T 5dc92680a7984bd09d8b942b7b2ad669 P 6824aa31d2ba4f3d9a643376b96be9aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:36.802896 5503 raft_consensus.cc:2258] T 5dc92680a7984bd09d8b942b7b2ad669 P 6824aa31d2ba4f3d9a643376b96be9aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:36.803227 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 5077
I20260924 00:19:36.803440 5503 ts_tablet_manager.cc:1905] T 5dc92680a7984bd09d8b942b7b2ad669 P 6824aa31d2ba4f3d9a643376b96be9aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:36.803835 5505 tablet_replica.cc:330] T 5dc92680a7984bd09d8b942b7b2ad669 P b26f2e09fb9049b5b11c7563e8c44c96: stopping tablet replica
I20260924 00:19:36.803936 5505 raft_consensus.cc:2229] T 5dc92680a7984bd09d8b942b7b2ad669 P b26f2e09fb9049b5b11c7563e8c44c96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:36.804028 5505 raft_consensus.cc:2258] T 5dc92680a7984bd09d8b942b7b2ad669 P b26f2e09fb9049b5b11c7563e8c44c96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:36.804243 5504 tablet_replica.cc:330] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac: stopping tablet replica
I20260924 00:19:36.804327 5504 raft_consensus.cc:2229] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:19:36.804414 5504 raft_consensus.cc:2258] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:36.805171 5505 ts_tablet_manager.cc:1905] T 5dc92680a7984bd09d8b942b7b2ad669 P b26f2e09fb9049b5b11c7563e8c44c96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:36.806598 5504 ts_tablet_manager.cc:1905] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:36.807082 5505 ts_tablet_manager.cc:1918] T 5dc92680a7984bd09d8b942b7b2ad669 P b26f2e09fb9049b5b11c7563e8c44c96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:19:36.807159 5505 log.cc:1196] T 5dc92680a7984bd09d8b942b7b2ad669 P b26f2e09fb9049b5b11c7563e8c44c96: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-1/wal/wals/5dc92680a7984bd09d8b942b7b2ad669
I20260924 00:19:36.807456 5505 ts_tablet_manager.cc:1939] T 5dc92680a7984bd09d8b942b7b2ad669 P b26f2e09fb9049b5b11c7563e8c44c96: Deleting consensus metadata
I20260924 00:19:36.807961 5004 catalog_manager.cc:4841] TS b26f2e09fb9049b5b11c7563e8c44c96 (127.0.98.2:33971): tablet 5dc92680a7984bd09d8b942b7b2ad669 (table test-table [id=311f9240f36f4f02bd9a35c48be5b667]) successfully deleted
I20260924 00:19:36.809301 5504 ts_tablet_manager.cc:1918] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:19:36.809474 5504 log.cc:1196] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/5dc92680a7984bd09d8b942b7b2ad669
W20260924 00:19:36.809690 5002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40711 (error 108)
I20260924 00:19:36.809983 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 5208
I20260924 00:19:36.810020 5504 ts_tablet_manager.cc:1939] T 5dc92680a7984bd09d8b942b7b2ad669 P 1a9a9ca7297c4ad985dcb23a2b44c3ac: Deleting consensus metadata
I20260924 00:19:36.810861 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.810162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40711 (local address 127.0.0.1:42536)
0924 00:19:36.810775 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:36.810823 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":590}
W20260924 00:19:36.812712 5002 catalog_manager.cc:4584] TS 6824aa31d2ba4f3d9a643376b96be9aa (127.0.98.1:40711): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5dc92680a7984bd09d8b942b7b2ad669: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40711: connect: Connection refused (error 111)
I20260924 00:19:36.816370 5004 catalog_manager.cc:4841] TS 1a9a9ca7297c4ad985dcb23a2b44c3ac (127.0.98.3:41979): tablet 5dc92680a7984bd09d8b942b7b2ad669 (table test-table [id=311f9240f36f4f02bd9a35c48be5b667]) successfully deleted
I20260924 00:19:36.816826 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 5339
I20260924 00:19:36.822202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 4986
2026-09-24T00:19:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (678 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 00:19:36.841810 392 mini_oidc.cc:152] Starting JWKS server
I20260924 00:19:36.842875 392 webserver.cc:460] Webserver started at https://127.0.0.1:44047/ using document root <none> and password file <none>
I20260924 00:19:36.843003 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 00:19:36.843192 392 webserver.cc:460] Webserver started at http://127.0.0.1:46731/ using document root <none> and password file <none>
2026-09-24T00:19:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:19:36Z Disabled control of system clock
I20260924 00:19:36.851101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41303
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41303
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44047/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790209154318147-392-0/testchainca.cert with env {}
W20260924 00:19:36.935614 5512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:36.935822 5512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:36.935845 5512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:36.936998 5512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:19:36.937037 5512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:36.937050 5512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:36.937062 5512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:19:36.937075 5512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 00:19:36.937088 5512 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 00:19:36.937098 5512 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44047/jwks.json
I20260924 00:19:36.938594 5512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44047/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790209154318147-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:36.938786 5512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:36.938947 5512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:36.943886 5517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:36.943876 5520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:36.943928 5518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:36.944322 5512 server_base.cc:1034] running on GCE node
I20260924 00:19:36.944619 5512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:36.944967 5512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:36.946161 5512 hybrid_clock.cc:648] HybridClock initialized: now 1790209176946156 us; error 48 us; skew 500 ppm
I20260924 00:19:36.947818 5512 webserver.cc:460] Webserver started at http://127.0.98.62:38689/ using document root <none> and password file <none>
I20260924 00:19:36.948139 5512 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:36.948241 5512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:36.948364 5512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:36.949499 5512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "650a639fe4944d4d8d14d7685f86e89d"
format_stamp: "Formatted at 2026-09-24 00:19:36 on dist-test-slave-777x"
server_key: "0f5b7e377c214d99935a442391f70cfb"
server_key_iv: "8c67155ee47d1db5366076cda335583b"
server_key_version: "encryptionkey@0"
I20260924 00:19:36.949837 5512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "650a639fe4944d4d8d14d7685f86e89d"
format_stamp: "Formatted at 2026-09-24 00:19:36 on dist-test-slave-777x"
server_key: "0f5b7e377c214d99935a442391f70cfb"
server_key_iv: "8c67155ee47d1db5366076cda335583b"
server_key_version: "encryptionkey@0"
I20260924 00:19:36.951133 5512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:19:36.951900 5526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:36.952114 5512 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:36.952245 5512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "650a639fe4944d4d8d14d7685f86e89d"
format_stamp: "Formatted at 2026-09-24 00:19:36 on dist-test-slave-777x"
server_key: "0f5b7e377c214d99935a442391f70cfb"
server_key_iv: "8c67155ee47d1db5366076cda335583b"
server_key_version: "encryptionkey@0"
I20260924 00:19:36.952327 5512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:36.962580 5512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:36.962883 5512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:36.963007 5512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:36.967073 5512 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41303
I20260924 00:19:36.967155 5578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41303 every 8 connection(s)
I20260924 00:19:36.967478 5512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:19:36.968608 5579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:36.970811 5579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d: Bootstrap starting.
I20260924 00:19:36.971352 5579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:36.971588 5579 log.cc:824] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:36.972373 5579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d: No bootstrap required, opened a new log
I20260924 00:19:36.973860 5579 raft_consensus.cc:348] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650a639fe4944d4d8d14d7685f86e89d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41303 } }
I20260924 00:19:36.974007 5579 raft_consensus.cc:374] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:36.974031 5579 raft_consensus.cc:729] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 650a639fe4944d4d8d14d7685f86e89d, State: Initialized, Role: FOLLOWER
I20260924 00:19:36.974135 5579 consensus_queue.cc:260] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650a639fe4944d4d8d14d7685f86e89d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41303 } }
I20260924 00:19:36.974217 5579 raft_consensus.cc:388] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:19:36.974254 5579 raft_consensus.cc:482] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:19:36.974334 5579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:36.974869 5579 raft_consensus.cc:504] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650a639fe4944d4d8d14d7685f86e89d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41303 } }
I20260924 00:19:36.974985 5579 leader_election.cc:302] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 650a639fe4944d4d8d14d7685f86e89d; no voters:
I20260924 00:19:36.975162 5579 leader_election.cc:288] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:19:36.975265 5582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:36.975426 5579 sys_catalog.cc:564] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:19:36.975489 5582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d [term 1 LEADER]: Becoming Leader. State: Replica: 650a639fe4944d4d8d14d7685f86e89d, State: Running, Role: LEADER
I20260924 00:19:36.975601 5582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650a639fe4944d4d8d14d7685f86e89d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41303 } }
I20260924 00:19:36.975764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 5512
I20260924 00:19:36.975872 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:19:36.976042 5584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 650a639fe4944d4d8d14d7685f86e89d. Latest consensus state: current_term: 1 leader_uuid: "650a639fe4944d4d8d14d7685f86e89d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650a639fe4944d4d8d14d7685f86e89d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41303 } } }
I20260924 00:19:36.976121 5584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:36.976154 392 external_mini_cluster.cc:1428] Setting key 2571541d560b67b3b9706e09bbdd26d1
I20260924 00:19:36.976123 5583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "650a639fe4944d4d8d14d7685f86e89d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "650a639fe4944d4d8d14d7685f86e89d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41303 } } }
I20260924 00:19:36.976379 5583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d [sys.catalog]: This master's current role is: LEADER
W20260924 00:19:36.977516 5600 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:19:36.977597 5600 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:19:36.977666 5591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:19:36.977861 5591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:19:36.979437 5591 catalog_manager.cc:1347] Generated new cluster ID: 821d800fc5ec4d96a624ec4e0b9e5a8d
I20260924 00:19:36.979496 5591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:19:36.982847 5599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:36.976754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59836 (local address 127.0.98.62:41303)
0924 00:19:36.977171 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:36.977182 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:36.977353 (+ 171us) server_negotiation.cc:407] Connection header received
0924 00:19:36.977481 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:36.977489 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:36.977561 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:36.977687 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:36.978004 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:36.979345 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:36.980524 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:36.981196 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:36.981332 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:36.982102 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:36.982106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:36.982107 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:19:36.982329 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:36.982372 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:19:36.982387 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:36.982455 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:36.982546 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 00:19:36.982626 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":104,"threads_started":1}
I20260924 00:19:36.983865 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:19:37.026535 5591 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:19:37.027176 5591 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:19:37.030817 5591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 650a639fe4944d4d8d14d7685f86e89d: Generated new TSK 0
I20260924 00:19:37.031045 5591 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 00:19:37.067941 5609 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 00:19:37.068210 5609 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 00:19:37.068290 5599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:37.045998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59840 (local address 127.0.98.62:41303)
0924 00:19:37.046051 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:37.046054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:37.046257 (+ 203us) server_negotiation.cc:407] Connection header received
0924 00:19:37.046338 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:37.046339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:37.046351 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:37.046414 (+ 63us) server_negotiation.cc:226] Negotiated authn=JWT
0924 00:19:37.046606 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:37.047119 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:37.048319 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:37.048529 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:37.048587 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:37.048819 (+ 232us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 00:19:37.067912 (+ 19093us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44047/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 00:19:37.068114 (+ 202us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59840: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44047/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 00:19:37.068357 5599 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59840: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44047/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 00:19:37.068585 5608 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:37.045913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41303 (local address 127.0.0.1:59840)
0924 00:19:37.046134 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:37.046146 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:37.046237 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:37.046443 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:37.046444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:37.046447 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0924 00:19:37.046533 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:37.046536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:37.047228 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:37.047229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:37.048142 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:37.048144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:37.048612 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:37.048616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:37.048684 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:37.048707 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 00:19:37.068082 (+ 19375us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44047/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 00:19:37.068239 (+ 157us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41303: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44047/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":98,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:41303: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44047/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 00:19:37.069187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 5512
2026-09-24T00:19:37Z chronyd exiting
I20260924 00:19:37.105146 392 test_util.cc:175] -----------------------------------------------
I20260924 00:19:37.105254 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790209154318147-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (264 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 00:19:37.105988 392 mini_oidc.cc:152] Starting JWKS server
I20260924 00:19:37.106923 392 webserver.cc:460] Webserver started at https://127.0.0.1:41105/ using document root <none> and password file <none>
I20260924 00:19:37.107009 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 00:19:37.107278 392 webserver.cc:460] Webserver started at http://127.0.0.1:40557/ using document root <none> and password file <none>
2026-09-24T00:19:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:19:37Z Disabled control of system clock
I20260924 00:19:37.114898 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41965
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41965
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41105/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790209154318147-392-0/testchainca.cert with env {}
W20260924 00:19:37.195972 5615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:37.196275 5615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:37.196348 5615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:37.197588 5615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:19:37.197661 5615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:37.197683 5615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:37.197705 5615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:19:37.197731 5615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 00:19:37.197760 5615 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 00:19:37.197784 5615 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41105/jwks.json
I20260924 00:19:37.199333 5615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41965
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41105/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790209154318147-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:37.199621 5615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:37.199844 5615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:37.205061 5623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:37.205060 5621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:37.205230 5620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:37.205523 5615 server_base.cc:1034] running on GCE node
I20260924 00:19:37.205806 5615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:37.206101 5615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:37.207248 5615 hybrid_clock.cc:648] HybridClock initialized: now 1790209177207230 us; error 38 us; skew 500 ppm
I20260924 00:19:37.208590 5615 webserver.cc:460] Webserver started at http://127.0.98.62:34571/ using document root <none> and password file <none>
I20260924 00:19:37.208877 5615 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:37.208930 5615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:37.209057 5615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:37.210122 5615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "752bdf66f62e488e9bdc7b5b4a922429"
format_stamp: "Formatted at 2026-09-24 00:19:37 on dist-test-slave-777x"
server_key: "7b6b2346a3c20e2d16f1d7cf58a37539"
server_key_iv: "c603fb9b27d3598ccdcd3cd2c9e61169"
server_key_version: "encryptionkey@0"
I20260924 00:19:37.210494 5615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "752bdf66f62e488e9bdc7b5b4a922429"
format_stamp: "Formatted at 2026-09-24 00:19:37 on dist-test-slave-777x"
server_key: "7b6b2346a3c20e2d16f1d7cf58a37539"
server_key_iv: "c603fb9b27d3598ccdcd3cd2c9e61169"
server_key_version: "encryptionkey@0"
I20260924 00:19:37.211831 5615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:19:37.212699 5629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:37.212924 5615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:19:37.213014 5615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "752bdf66f62e488e9bdc7b5b4a922429"
format_stamp: "Formatted at 2026-09-24 00:19:37 on dist-test-slave-777x"
server_key: "7b6b2346a3c20e2d16f1d7cf58a37539"
server_key_iv: "c603fb9b27d3598ccdcd3cd2c9e61169"
server_key_version: "encryptionkey@0"
I20260924 00:19:37.213107 5615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:37.234385 5615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:37.234750 5615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:37.234915 5615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:37.239017 5615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41965
I20260924 00:19:37.239054 5681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41965 every 8 connection(s)
I20260924 00:19:37.239455 5615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:19:37.240470 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:37.242810 5682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429: Bootstrap starting.
I20260924 00:19:37.243438 5682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:37.243726 5682 log.cc:824] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:37.244469 5682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429: No bootstrap required, opened a new log
I20260924 00:19:37.246083 5682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752bdf66f62e488e9bdc7b5b4a922429" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } }
I20260924 00:19:37.246225 5682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:37.246248 5682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 752bdf66f62e488e9bdc7b5b4a922429, State: Initialized, Role: FOLLOWER
I20260924 00:19:37.246439 5682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752bdf66f62e488e9bdc7b5b4a922429" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } }
I20260924 00:19:37.246524 5682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:19:37.246567 5682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:19:37.246624 5682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:37.247143 5682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752bdf66f62e488e9bdc7b5b4a922429" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } }
I20260924 00:19:37.247254 5682 leader_election.cc:302] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 752bdf66f62e488e9bdc7b5b4a922429; no voters:
I20260924 00:19:37.247437 5682 leader_election.cc:288] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:19:37.247481 5685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:37.247649 5685 raft_consensus.cc:686] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429 [term 1 LEADER]: Becoming Leader. State: Replica: 752bdf66f62e488e9bdc7b5b4a922429, State: Running, Role: LEADER
I20260924 00:19:37.247716 5682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:19:37.247741 5685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752bdf66f62e488e9bdc7b5b4a922429" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } }
I20260924 00:19:37.248137 5686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "752bdf66f62e488e9bdc7b5b4a922429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752bdf66f62e488e9bdc7b5b4a922429" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } } }
I20260924 00:19:37.248255 5686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:37.248322 5687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 752bdf66f62e488e9bdc7b5b4a922429. Latest consensus state: current_term: 1 leader_uuid: "752bdf66f62e488e9bdc7b5b4a922429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752bdf66f62e488e9bdc7b5b4a922429" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41965 } } }
I20260924 00:19:37.248407 5687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:37.248482 5692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:19:37.249037 5692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:19:37.249513 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 5615
I20260924 00:19:37.249614 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:19:37.249796 392 external_mini_cluster.cc:1428] Setting key 5141096c89e824073cdbfde572895f13
I20260924 00:19:37.250554 5692 catalog_manager.cc:1347] Generated new cluster ID: 28669625e07742358bb106c7cffcf111
I20260924 00:19:37.250617 5692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:19:37.256372 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:37.250427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43098 (local address 127.0.98.62:41965)
0924 00:19:37.250616 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:37.250626 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:37.250657 (+ 31us) server_negotiation.cc:407] Connection header received
0924 00:19:37.250772 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:37.250779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:37.250846 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:37.250968 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:37.251292 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:37.252669 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:37.253762 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:37.254412 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:37.254515 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:37.255468 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:37.255473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:37.255474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:19:37.255677 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:37.255717 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:19:37.255731 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:37.255783 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:37.256033 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 00:19:37.256121 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260924 00:19:37.257232 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:19:37.261672 5692 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:19:37.262223 5692 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:19:37.264447 5692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 752bdf66f62e488e9bdc7b5b4a922429: Generated new TSK 0
I20260924 00:19:37.264645 5692 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 00:19:37.296914 5712 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 00:19:37.297008 5712 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 00:19:37.297355 5705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:37.274789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43102 (local address 127.0.98.62:41965)
0924 00:19:37.274855 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:37.274857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:37.274977 (+ 120us) server_negotiation.cc:407] Connection header received
0924 00:19:37.275054 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:37.275055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:37.275067 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:37.275134 (+ 67us) server_negotiation.cc:226] Negotiated authn=JWT
0924 00:19:37.275297 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:37.275808 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:37.277110 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:37.277376 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:37.277437 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:37.277636 (+ 199us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 00:19:37.297123 (+ 19487us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 00:19:37.297297 (+ 174us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43102: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 00:19:37.297434 5705 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43102: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 00:19:37.297497 5711 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:37.274713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41965 (local address 127.0.0.1:43102)
0924 00:19:37.274899 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:37.274911 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:37.274982 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:37.275151 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:37.275152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:37.275155 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0924 00:19:37.275218 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:37.275220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:37.275900 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:37.275901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:37.277006 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:37.277010 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:37.277454 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:37.277459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:37.277526 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:37.277532 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 00:19:37.297278 (+ 19746us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 00:19:37.297399 (+ 121us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41965: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41105/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":103,"threads_started":1}
I20260924 00:19:37.297866 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 5615
2026-09-24T00:19:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (229 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 00:19:37.335520 392 mini_oidc.cc:152] Starting JWKS server
I20260924 00:19:37.336326 392 webserver.cc:460] Webserver started at https://127.0.0.1:46271/ using document root <none> and password file <none>
I20260924 00:19:37.336386 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 00:19:37.336512 392 webserver.cc:460] Webserver started at http://127.0.0.1:43767/ using document root <none> and password file <none>
2026-09-24T00:19:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:19:37Z Disabled control of system clock
I20260924 00:19:37.344834 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34449
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34449
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46271/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 00:19:37.426209 5718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:37.426421 5718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:37.426442 5718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:37.427544 5718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:19:37.427585 5718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:37.427598 5718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:37.427610 5718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:19:37.427623 5718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 00:19:37.427635 5718 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 00:19:37.427646 5718 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46271/jwks.json
I20260924 00:19:37.429044 5718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46271/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:37.429251 5718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:37.429489 5718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:37.434449 5724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:37.434449 5726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:37.434634 5718 server_base.cc:1034] running on GCE node
W20260924 00:19:37.434527 5723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:37.434935 5718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:37.435276 5718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:37.436424 5718 hybrid_clock.cc:648] HybridClock initialized: now 1790209177436409 us; error 28 us; skew 500 ppm
I20260924 00:19:37.437810 5718 webserver.cc:460] Webserver started at http://127.0.98.62:40365/ using document root <none> and password file <none>
I20260924 00:19:37.438082 5718 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:37.438167 5718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:37.438318 5718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:37.439510 5718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "1cc896f01cc3495183b386557cd502c1"
format_stamp: "Formatted at 2026-09-24 00:19:37 on dist-test-slave-777x"
server_key: "996118ea177d64b00b45160306c03e84"
server_key_iv: "3b3c91d1f0909e09879c21ab435598dc"
server_key_version: "encryptionkey@0"
I20260924 00:19:37.439910 5718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "1cc896f01cc3495183b386557cd502c1"
format_stamp: "Formatted at 2026-09-24 00:19:37 on dist-test-slave-777x"
server_key: "996118ea177d64b00b45160306c03e84"
server_key_iv: "3b3c91d1f0909e09879c21ab435598dc"
server_key_version: "encryptionkey@0"
I20260924 00:19:37.441385 5718 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:19:37.442139 5732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:37.442360 5718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:37.442468 5718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "1cc896f01cc3495183b386557cd502c1"
format_stamp: "Formatted at 2026-09-24 00:19:37 on dist-test-slave-777x"
server_key: "996118ea177d64b00b45160306c03e84"
server_key_iv: "3b3c91d1f0909e09879c21ab435598dc"
server_key_version: "encryptionkey@0"
I20260924 00:19:37.442553 5718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:37.453202 5718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:37.453523 5718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:37.453670 5718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:37.457831 5718 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34449
I20260924 00:19:37.457926 5784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34449 every 8 connection(s)
I20260924 00:19:37.458279 5718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:19:37.458962 5785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:37.459406 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 5718
I20260924 00:19:37.459545 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:19:37.459730 392 external_mini_cluster.cc:1428] Setting key b34b32c03d574e9a216f3c292cea14ae
I20260924 00:19:37.463156 5785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1: Bootstrap starting.
I20260924 00:19:37.463805 5785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:37.464118 5785 log.cc:824] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:37.464890 5785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1: No bootstrap required, opened a new log
I20260924 00:19:37.466151 5788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:37.460290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35608 (local address 127.0.98.62:34449)
0924 00:19:37.460521 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:37.460529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:37.460564 (+ 35us) server_negotiation.cc:407] Connection header received
0924 00:19:37.461193 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:37.461202 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:37.461290 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:37.461456 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:37.461668 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:37.463012 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:37.464017 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:37.464538 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:37.464695 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:37.465462 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:37.465466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:37.465467 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:19:37.465680 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:37.465716 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:19:37.465728 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:37.465800 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:37.465896 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 00:19:37.465982 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":94,"threads_started":1}
I20260924 00:19:37.467121 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:19:37.467032 5785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc896f01cc3495183b386557cd502c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34449 } }
I20260924 00:19:37.467252 5785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:37.467307 5785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc896f01cc3495183b386557cd502c1, State: Initialized, Role: FOLLOWER
I20260924 00:19:37.467437 5785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc896f01cc3495183b386557cd502c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34449 } }
I20260924 00:19:37.467538 5785 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:19:37.467585 5785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:19:37.467643 5785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:37.468452 5785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc896f01cc3495183b386557cd502c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34449 } }
I20260924 00:19:37.468581 5785 leader_election.cc:302] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cc896f01cc3495183b386557cd502c1; no voters:
I20260924 00:19:37.468761 5785 leader_election.cc:288] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:19:37.468842 5790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:37.469022 5790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc896f01cc3495183b386557cd502c1, State: Running, Role: LEADER
I20260924 00:19:37.469095 5785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:19:37.469107 5790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc896f01cc3495183b386557cd502c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34449 } }
I20260924 00:19:37.469553 5791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cc896f01cc3495183b386557cd502c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc896f01cc3495183b386557cd502c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34449 } } }
I20260924 00:19:37.469671 5791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:37.469530 5792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cc896f01cc3495183b386557cd502c1. Latest consensus state: current_term: 1 leader_uuid: "1cc896f01cc3495183b386557cd502c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc896f01cc3495183b386557cd502c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34449 } } }
I20260924 00:19:37.469835 5792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:37.470173 5798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:19:37.470777 5798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:19:37.472509 5798 catalog_manager.cc:1347] Generated new cluster ID: c8ac5715c4df442eaf48b22289e6d780
I20260924 00:19:37.472577 5798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:19:37.492653 5798 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:19:37.493482 5798 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:19:37.498870 5798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cc896f01cc3495183b386557cd502c1: Generated new TSK 0
I20260924 00:19:37.499078 5798 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:19:37.518059 5788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:37.514004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35610 (local address 127.0.98.62:34449)
0924 00:19:37.514066 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:37.514069 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:37.514181 (+ 112us) server_negotiation.cc:407] Connection header received
0924 00:19:37.514272 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:37.514273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:37.514288 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:37.514366 (+ 78us) server_negotiation.cc:226] Negotiated authn=JWT
0924 00:19:37.514533 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:37.515329 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:37.516260 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:37.516587 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:37.516658 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:37.516855 (+ 197us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 00:19:37.517838 (+ 983us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46271/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 00:19:37.517979 (+ 141us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35610: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46271/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 00:19:37.518127 5788 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35610: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46271/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 00:19:37.520053 5814 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:37.513876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34449 (local address 127.0.0.1:35610)
0924 00:19:37.514096 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:37.514108 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:37.514186 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:37.514404 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:37.514405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:37.514407 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 00:19:37.514460 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:37.514462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:37.515444 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:37.515445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:37.516107 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:37.516109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:37.516684 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:37.516688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:37.516734 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:37.516739 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 00:19:37.517962 (+ 1223us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46271/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 00:19:37.519943 (+ 1981us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34449: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46271/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":1768,"thread_start_us":171,"threads_started":1}
I20260924 00:19:37.520257 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 5718
E20260924 00:19:37.529968 5815 webserver.cc:533] Webserver: sslize failed:
E20260924 00:19:37.530050 5815 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T00:19:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (222 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T00:19:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:19:37Z Disabled control of system clock
I20260924 00:19:37.565910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37589
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37589
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790209154318147-392-0/testchainca.cert with env {}
W20260924 00:19:37.646286 5819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:37.646512 5819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:37.646535 5819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:37.647681 5819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:19:37.647720 5819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:37.647733 5819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:37.647746 5819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:19:37.647758 5819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 00:19:37.647770 5819 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 00:19:37.647781 5819 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 00:19:37.649235 5819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37589
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790209154318147-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:37.649426 5819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:37.649588 5819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:37.654835 5825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:37.654822 5824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:37.654822 5827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:37.655153 5819 server_base.cc:1034] running on GCE node
I20260924 00:19:37.655437 5819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:37.655705 5819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:37.656862 5819 hybrid_clock.cc:648] HybridClock initialized: now 1790209177656783 us; error 89 us; skew 500 ppm
I20260924 00:19:37.658224 5819 webserver.cc:460] Webserver started at http://127.0.98.62:40815/ using document root <none> and password file <none>
I20260924 00:19:37.658514 5819 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:37.658565 5819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:37.658718 5819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:37.659823 5819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "0013757fc0bd4ac290f1103f3ac24840"
format_stamp: "Formatted at 2026-09-24 00:19:37 on dist-test-slave-777x"
server_key: "7195e0d8a3a35d6caf0586e09ccb497a"
server_key_iv: "d3e16beec34359a09364650f62286f83"
server_key_version: "encryptionkey@0"
I20260924 00:19:37.660259 5819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "0013757fc0bd4ac290f1103f3ac24840"
format_stamp: "Formatted at 2026-09-24 00:19:37 on dist-test-slave-777x"
server_key: "7195e0d8a3a35d6caf0586e09ccb497a"
server_key_iv: "d3e16beec34359a09364650f62286f83"
server_key_version: "encryptionkey@0"
I20260924 00:19:37.661628 5819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:37.662464 5833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:37.662708 5819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:37.662822 5819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "0013757fc0bd4ac290f1103f3ac24840"
format_stamp: "Formatted at 2026-09-24 00:19:37 on dist-test-slave-777x"
server_key: "7195e0d8a3a35d6caf0586e09ccb497a"
server_key_iv: "d3e16beec34359a09364650f62286f83"
server_key_version: "encryptionkey@0"
I20260924 00:19:37.662889 5819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:37.672012 5819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:37.672392 5819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:37.672529 5819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:37.677222 5819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37589
I20260924 00:19:37.677335 5885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37589 every 8 connection(s)
I20260924 00:19:37.677724 5819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:19:37.678812 5886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:37.680441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 5819
I20260924 00:19:37.680567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:19:37.680783 392 external_mini_cluster.cc:1428] Setting key 5bbfcaf289897746852faccab6e16350
I20260924 00:19:37.681764 5886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840: Bootstrap starting.
I20260924 00:19:37.682476 5886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:37.682812 5886 log.cc:824] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:37.683615 5886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840: No bootstrap required, opened a new log
I20260924 00:19:37.685832 5886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0013757fc0bd4ac290f1103f3ac24840" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37589 } }
I20260924 00:19:37.686034 5886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:37.686079 5886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0013757fc0bd4ac290f1103f3ac24840, State: Initialized, Role: FOLLOWER
I20260924 00:19:37.686208 5886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0013757fc0bd4ac290f1103f3ac24840" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37589 } }
I20260924 00:19:37.686331 5886 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:19:37.686383 5886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:19:37.686455 5886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:37.687248 5886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0013757fc0bd4ac290f1103f3ac24840" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37589 } }
I20260924 00:19:37.687366 5886 leader_election.cc:302] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0013757fc0bd4ac290f1103f3ac24840; no voters:
I20260924 00:19:37.687417 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:37.681430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48074 (local address 127.0.98.62:37589)
0924 00:19:37.681671 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:37.681680 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:37.681712 (+ 32us) server_negotiation.cc:407] Connection header received
0924 00:19:37.682281 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:37.682290 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:37.682374 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:37.682532 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:37.682833 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:37.684158 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:37.684950 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:37.685650 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:37.685779 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:37.686605 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:37.686609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:37.686610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:19:37.686820 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:37.686864 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:19:37.686880 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:37.686950 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:37.687086 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 00:19:37.687170 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":98,"threads_started":1}
I20260924 00:19:37.687525 5886 leader_election.cc:288] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:19:37.687731 5892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:37.687811 5886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:19:37.687870 5892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840 [term 1 LEADER]: Becoming Leader. State: Replica: 0013757fc0bd4ac290f1103f3ac24840, State: Running, Role: LEADER
I20260924 00:19:37.688067 5892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0013757fc0bd4ac290f1103f3ac24840" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37589 } }
I20260924 00:19:37.688491 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:19:37.688477 5893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0013757fc0bd4ac290f1103f3ac24840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0013757fc0bd4ac290f1103f3ac24840" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37589 } } }
I20260924 00:19:37.688563 5893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:37.688711 5891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0013757fc0bd4ac290f1103f3ac24840. Latest consensus state: current_term: 1 leader_uuid: "0013757fc0bd4ac290f1103f3ac24840" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0013757fc0bd4ac290f1103f3ac24840" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37589 } } }
I20260924 00:19:37.688757 5900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:19:37.688784 5891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:37.689580 5900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:19:37.691126 5900 catalog_manager.cc:1347] Generated new cluster ID: ec7701c190c84edca49b57a5bcf13166
I20260924 00:19:37.691190 5900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:19:37.730382 5900 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:19:37.731143 5900 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:19:37.739171 5900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0013757fc0bd4ac290f1103f3ac24840: Generated new TSK 0
I20260924 00:19:37.739393 5900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:19:37.874423 5889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:37.753321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48082 (local address 127.0.98.62:37589)
0924 00:19:37.753400 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:37.753403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:37.753592 (+ 189us) server_negotiation.cc:407] Connection header received
0924 00:19:37.753632 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:37.753633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:37.753645 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:37.753727 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0924 00:19:37.753885 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:37.754362 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:37.755423 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:37.755612 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:37.755671 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:37.755876 (+ 205us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 00:19:37.874164 (+118288us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 00:19:37.874362 (+ 198us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48082: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 00:19:37.874497 5889 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48082: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 00:19:37.874684 5915 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:37.753269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37589 (local address 127.0.0.1:48082)
0924 00:19:37.753469 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:37.753482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:37.753561 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:37.753744 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:37.753745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:37.753747 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 00:19:37.753819 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:37.753821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:37.754445 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:37.754446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:37.755322 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:37.755325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:37.755700 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:37.755704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:37.755770 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:37.755776 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 00:19:37.874368 (+118592us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 00:19:37.874566 (+ 198us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37589: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":131,"threads_started":1}
I20260924 00:19:37.875123 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 5819
2026-09-24T00:19:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (335 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (90 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:19:38 dist-test-slave-777x krb5kdc[5927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:19:38 dist-test-slave-777x krb5kdc[5927](info): set up 2 sockets
Sep 24 00:19:38 dist-test-slave-777x krb5kdc[5927](info): commencing operation
krb5kdc: starting...
W20260924 00:19:39.129026 392 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:19:39 dist-test-slave-777x krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:19:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:19:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:19:39.260469 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44063
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44063
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:39.346144 5940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:39.346380 5940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:39.346405 5940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:39.347523 5940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:19:39.347559 5940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:39.347573 5940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:39.347585 5940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:19:39.347597 5940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:19:39.349182 5940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:39.349397 5940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:39.349570 5940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:39.354889 5946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:39.354889 5945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:39.355000 5940 server_base.cc:1034] running on GCE node
W20260924 00:19:39.354889 5948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:39.355391 5940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:39.355707 5940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:39.356900 5940 hybrid_clock.cc:648] HybridClock initialized: now 1790209179356886 us; error 34 us; skew 500 ppm
Sep 24 00:19:39 dist-test-slave-777x krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:39.359786 5940 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:19:39.360402 5940 webserver.cc:460] Webserver started at http://127.0.98.62:43917/ using document root <none> and password file <none>
I20260924 00:19:39.360659 5940 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:39.360738 5940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:39.360898 5940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:39.362001 5940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "b60ede8bddbe4ce2a756e4d9bf6f4b04"
format_stamp: "Formatted at 2026-09-24 00:19:39 on dist-test-slave-777x"
server_key: "fe1c440a3e6969f65cf8272a348d5d31"
server_key_iv: "39f638d777cc0c16660036a17aad1516"
server_key_version: "encryptionkey@0"
I20260924 00:19:39.362387 5940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "b60ede8bddbe4ce2a756e4d9bf6f4b04"
format_stamp: "Formatted at 2026-09-24 00:19:39 on dist-test-slave-777x"
server_key: "fe1c440a3e6969f65cf8272a348d5d31"
server_key_iv: "39f638d777cc0c16660036a17aad1516"
server_key_version: "encryptionkey@0"
I20260924 00:19:39.363787 5940 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:19:39.364815 5955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:39.365069 5940 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:19:39.365181 5940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "b60ede8bddbe4ce2a756e4d9bf6f4b04"
format_stamp: "Formatted at 2026-09-24 00:19:39 on dist-test-slave-777x"
server_key: "fe1c440a3e6969f65cf8272a348d5d31"
server_key_iv: "39f638d777cc0c16660036a17aad1516"
server_key_version: "encryptionkey@0"
I20260924 00:19:39.365268 5940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:39.399174 5940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:39.401978 5940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:39.402195 5940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:39.406936 5940 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44063
I20260924 00:19:39.407353 6007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44063 every 8 connection(s)
I20260924 00:19:39.407466 5940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:19:39.408417 6008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:39.410703 6008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04: Bootstrap starting.
I20260924 00:19:39.411242 6008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:39.411484 6008 log.cc:824] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:39.412353 6008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04: No bootstrap required, opened a new log
I20260924 00:19:39.414137 6008 raft_consensus.cc:348] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60ede8bddbe4ce2a756e4d9bf6f4b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44063 } }
I20260924 00:19:39.414294 6008 raft_consensus.cc:374] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:39.414320 6008 raft_consensus.cc:729] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b60ede8bddbe4ce2a756e4d9bf6f4b04, State: Initialized, Role: FOLLOWER
I20260924 00:19:39.414446 6008 consensus_queue.cc:260] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60ede8bddbe4ce2a756e4d9bf6f4b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44063 } }
I20260924 00:19:39.414513 6008 raft_consensus.cc:388] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:19:39.414536 6008 raft_consensus.cc:482] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:19:39.414572 6008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:39.415472 6008 raft_consensus.cc:504] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60ede8bddbe4ce2a756e4d9bf6f4b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44063 } }
I20260924 00:19:39.415566 6008 leader_election.cc:302] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b60ede8bddbe4ce2a756e4d9bf6f4b04; no voters:
I20260924 00:19:39.415709 6008 leader_election.cc:288] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:19:39.415772 6011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:39.415949 6011 raft_consensus.cc:686] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04 [term 1 LEADER]: Becoming Leader. State: Replica: b60ede8bddbe4ce2a756e4d9bf6f4b04, State: Running, Role: LEADER
I20260924 00:19:39.416085 6011 consensus_queue.cc:237] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60ede8bddbe4ce2a756e4d9bf6f4b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44063 } }
I20260924 00:19:39.416198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 5940
I20260924 00:19:39.416339 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:19:39.416445 6008 sys_catalog.cc:564] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:19:39.416570 392 external_mini_cluster.cc:1428] Setting key d4366e20144343dc76d20d001ea7771b
I20260924 00:19:39.416522 6012 sys_catalog.cc:455] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b60ede8bddbe4ce2a756e4d9bf6f4b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60ede8bddbe4ce2a756e4d9bf6f4b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44063 } } }
I20260924 00:19:39.416534 6013 sys_catalog.cc:455] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b60ede8bddbe4ce2a756e4d9bf6f4b04. Latest consensus state: current_term: 1 leader_uuid: "b60ede8bddbe4ce2a756e4d9bf6f4b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60ede8bddbe4ce2a756e4d9bf6f4b04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44063 } } }
I20260924 00:19:39.416608 6012 sys_catalog.cc:458] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:39.416630 6013 sys_catalog.cc:458] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:39.416842 6015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:19:39.417590 6015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:19:39.419296 6015 catalog_manager.cc:1347] Generated new cluster ID: 66083db32b4e4c02946f8887648c1823
I20260924 00:19:39.419350 6015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:19:39 dist-test-slave-777x krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:39.428135 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:39.417294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.0.98.62:44063)
0924 00:19:39.417791 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:39.417805 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:39.417849 (+ 44us) server_negotiation.cc:407] Connection header received
0924 00:19:39.417998 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:39.418006 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:39.418077 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:39.418328 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:39.419083 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:39.420241 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:39.421112 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:39.421603 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:39.421745 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:39.424396 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:39.424400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:39.424401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:39.424415 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:39.426318 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:39.426743 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:39.426747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:39.426747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:39.426859 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:39.427109 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:39.427110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:39.427110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:39.427378 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:39.427582 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:39.427660 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 00:19:39.427827 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":90,"threads_started":1}
I20260924 00:19:39.428778 6015 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:19:39.429515 6015 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:19:39.434131 6015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b60ede8bddbe4ce2a756e4d9bf6f4b04: Generated new TSK 0
I20260924 00:19:39.434404 6015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:19:39.464591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44063
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:39.555262 6034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:39.555603 6034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:39.555689 6034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:39.557129 6034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:39.557204 6034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:39.557253 6034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:19:39.558825 6034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:39.559130 6034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:39.559438 6034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:39.565164 6042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:39.565268 6040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:39.565351 6039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:39.565753 6034 server_base.cc:1034] running on GCE node
I20260924 00:19:39.566027 6034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:39.566313 6034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:39.567520 6034 hybrid_clock.cc:648] HybridClock initialized: now 1790209179567508 us; error 34 us; skew 500 ppm
Sep 24 00:19:39 dist-test-slave-777x krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:39.570638 6034 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:19:39.571200 6034 webserver.cc:460] Webserver started at http://127.0.98.1:35683/ using document root <none> and password file <none>
I20260924 00:19:39.571466 6034 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:39.571513 6034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:39.571659 6034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:39.572913 6034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "32efb8e16a6c45609286ffaf334562fd"
format_stamp: "Formatted at 2026-09-24 00:19:39 on dist-test-slave-777x"
server_key: "a5a40385b3e30c7180f70575ba44467a"
server_key_iv: "513510b9c5ddc906972b5e2e219780a3"
server_key_version: "encryptionkey@0"
I20260924 00:19:39.573295 6034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "32efb8e16a6c45609286ffaf334562fd"
format_stamp: "Formatted at 2026-09-24 00:19:39 on dist-test-slave-777x"
server_key: "a5a40385b3e30c7180f70575ba44467a"
server_key_iv: "513510b9c5ddc906972b5e2e219780a3"
server_key_version: "encryptionkey@0"
I20260924 00:19:39.574790 6034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:19:39.575641 6049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:39.575970 6034 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:39.576092 6034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "32efb8e16a6c45609286ffaf334562fd"
format_stamp: "Formatted at 2026-09-24 00:19:39 on dist-test-slave-777x"
server_key: "a5a40385b3e30c7180f70575ba44467a"
server_key_iv: "513510b9c5ddc906972b5e2e219780a3"
server_key_version: "encryptionkey@0"
I20260924 00:19:39.576241 6034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:39.589522 6034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:39.592103 6034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:39.592360 6034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:39.592602 6034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:39.592990 6034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:39.593058 6034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:39.593112 6034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:39.593142 6034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:39.600574 6034 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42651
I20260924 00:19:39.600613 6162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42651 every 8 connection(s)
I20260924 00:19:39.601011 6034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 00:19:39 dist-test-slave-777x krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:39.610217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 6034
I20260924 00:19:39.610308 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:19:39.610517 392 external_mini_cluster.cc:1428] Setting key 8f8e29af99c9265baadd2f5f906e6c50
I20260924 00:19:39.612390 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:39.601871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39267 (local address 127.0.98.62:44063)
0924 00:19:39.601982 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:39.601984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:39.602284 (+ 300us) server_negotiation.cc:407] Connection header received
0924 00:19:39.603362 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:39.603364 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:39.603381 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:39.603452 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:39.604333 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:39.604920 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:39.606123 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:39.606426 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:39.606531 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:39.609049 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:39.609063 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:39.609064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:39.609071 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:39.610884 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:39.611362 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:39.611364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:39.611364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:39.611424 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:39.611692 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:39.611693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:39.611694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:39.611816 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:39.611893 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:39.612206 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 00:19:39.612295 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 00:19:39.612617 6165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:39.601822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44063 (local address 127.0.98.1:39267)
0924 00:19:39.602023 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:39.602072 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:39.603205 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:39.603536 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:39.603540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:39.603856 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:39.604081 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:39.604086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:39.605057 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:39.605058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:39.605999 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:39.606001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:39.606544 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:39.606545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:39.606696 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:39.607230 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:39.607239 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:39.608934 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:39.610980 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:39.610988 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:39.610991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:39.611297 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:39.611496 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:39.611497 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:39.611497 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:39.611618 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:39.611883 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:39.611891 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:39.612041 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 00:19:39.612158 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 00:19:39.612364 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":90,"threads_started":1}
I20260924 00:19:39.613168 6163 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44063
I20260924 00:19:39.613269 6163 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:39.613451 6163 heartbeater.cc:511] Master 127.0.98.62:44063 requested a full tablet report, sending...
I20260924 00:19:39.614059 5972 ts_manager.cc:194] Registered new tserver with Master: 32efb8e16a6c45609286ffaf334562fd (127.0.98.1:42651)
I20260924 00:19:39.614990 5972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39267
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:19:39.639851 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44063
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:39.723206 6168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:39.723435 6168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:39.723459 6168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:39.724872 6168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:39.724931 6168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:39.724960 6168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:19:39.726526 6168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:39.726850 6168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:39.727113 6168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:39.732683 6174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:39.732592 6173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:39.732636 6168 server_base.cc:1034] running on GCE node
W20260924 00:19:39.732662 6176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:39.733094 6168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:39.733363 6168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:39.734529 6168 hybrid_clock.cc:648] HybridClock initialized: now 1790209179734526 us; error 38 us; skew 500 ppm
Sep 24 00:19:39 dist-test-slave-777x krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:39.737340 6168 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:19:39.737820 6168 webserver.cc:460] Webserver started at http://127.0.98.2:40613/ using document root <none> and password file <none>
I20260924 00:19:39.738029 6168 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:39.738075 6168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:39.738169 6168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:39.739182 6168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "d5a4469e043f47b9a99961b0c78767cc"
format_stamp: "Formatted at 2026-09-24 00:19:39 on dist-test-slave-777x"
server_key: "cd0ae0c1c5f6f15854db123cc077f02b"
server_key_iv: "36598636467267768e7224f20145f192"
server_key_version: "encryptionkey@0"
I20260924 00:19:39.739509 6168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d5a4469e043f47b9a99961b0c78767cc"
format_stamp: "Formatted at 2026-09-24 00:19:39 on dist-test-slave-777x"
server_key: "cd0ae0c1c5f6f15854db123cc077f02b"
server_key_iv: "36598636467267768e7224f20145f192"
server_key_version: "encryptionkey@0"
I20260924 00:19:39.740860 6168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 00:19:39.741628 6183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:39.741847 6168 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:39.741930 6168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "d5a4469e043f47b9a99961b0c78767cc"
format_stamp: "Formatted at 2026-09-24 00:19:39 on dist-test-slave-777x"
server_key: "cd0ae0c1c5f6f15854db123cc077f02b"
server_key_iv: "36598636467267768e7224f20145f192"
server_key_version: "encryptionkey@0"
I20260924 00:19:39.741989 6168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:39.753790 6168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:39.755781 6168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:39.755903 6168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:39.756088 6168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:39.756423 6168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:39.756460 6168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:39.756484 6168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:39.756495 6168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:39.762127 6168 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44145
I20260924 00:19:39.762202 6296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44145 every 8 connection(s)
I20260924 00:19:39.762532 6168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:19:39.764998 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 6168
I20260924 00:19:39.765117 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:19:39.765319 392 external_mini_cluster.cc:1428] Setting key e720caebefdcdb727ef13816ea5dda01
Sep 24 00:19:39 dist-test-slave-777x krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:39.773181 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:39.763282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41675 (local address 127.0.98.62:44063)
0924 00:19:39.763332 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:39.763335 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:39.763505 (+ 170us) server_negotiation.cc:407] Connection header received
0924 00:19:39.764599 (+ 1094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:39.764600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:39.764614 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:39.764665 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:39.765569 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:39.766183 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:39.767282 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:39.767583 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:39.767649 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:39.770233 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:39.770235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:39.770235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:39.770243 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:39.771754 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:39.772201 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:39.772202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:39.772203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:39.772262 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:39.772519 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:39.772520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:39.772521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:39.772641 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:39.772718 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:39.773005 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 00:19:39.773091 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 00:19:39.773449 6299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:39.763242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44063 (local address 127.0.98.2:41675)
0924 00:19:39.763374 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:39.763423 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:39.764475 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:39.764742 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:39.764746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:39.765178 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:39.765486 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:39.765492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:39.766297 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:39.766299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:39.767181 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:39.767184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:39.767669 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:39.767670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:39.767826 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:39.768432 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:39.768443 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:39.770099 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:39.771837 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:39.771844 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:39.771847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:39.772081 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:39.772340 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:39.772340 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:39.772341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:39.772449 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:39.772708 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:39.772714 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:39.772864 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 00:19:39.772973 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 00:19:39.773136 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 00:19:39.774147 6297 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44063
I20260924 00:19:39.774241 6297 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:39.774394 6297 heartbeater.cc:511] Master 127.0.98.62:44063 requested a full tablet report, sending...
I20260924 00:19:39.774746 5972 ts_manager.cc:194] Registered new tserver with Master: d5a4469e043f47b9a99961b0c78767cc (127.0.98.2:44145)
I20260924 00:19:39.775153 5972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:19:39.792886 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44063
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:39.875885 6302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:39.876089 6302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:39.876112 6302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:39.877395 6302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:39.877436 6302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:39.877471 6302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:19:39.878938 6302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:39.879160 6302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:39.879451 6302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:39.884621 6308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:39.884621 6310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:39.884785 6302 server_base.cc:1034] running on GCE node
W20260924 00:19:39.884840 6307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:39.885114 6302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:39.885375 6302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:39.886533 6302 hybrid_clock.cc:648] HybridClock initialized: now 1790209179886530 us; error 38 us; skew 500 ppm
Sep 24 00:19:39 dist-test-slave-777x krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:39.889231 6302 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:19:39.889689 6302 webserver.cc:460] Webserver started at http://127.0.98.3:35287/ using document root <none> and password file <none>
I20260924 00:19:39.889879 6302 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:39.889919 6302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:39.890008 6302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:39.891036 6302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "a86d0976e99943f7871b599bfdcbcaf2"
format_stamp: "Formatted at 2026-09-24 00:19:39 on dist-test-slave-777x"
server_key: "684800f4643703d1ee32f763ea25f7e7"
server_key_iv: "3b89e31ccae228817a0e691ba5602be2"
server_key_version: "encryptionkey@0"
I20260924 00:19:39.891345 6302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a86d0976e99943f7871b599bfdcbcaf2"
format_stamp: "Formatted at 2026-09-24 00:19:39 on dist-test-slave-777x"
server_key: "684800f4643703d1ee32f763ea25f7e7"
server_key_iv: "3b89e31ccae228817a0e691ba5602be2"
server_key_version: "encryptionkey@0"
I20260924 00:19:39.892680 6302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 00:19:39.893451 6317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:39.893641 6302 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:39.893733 6302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "a86d0976e99943f7871b599bfdcbcaf2"
format_stamp: "Formatted at 2026-09-24 00:19:39 on dist-test-slave-777x"
server_key: "684800f4643703d1ee32f763ea25f7e7"
server_key_iv: "3b89e31ccae228817a0e691ba5602be2"
server_key_version: "encryptionkey@0"
I20260924 00:19:39.893805 6302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:39.929049 6302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:39.931176 6302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:39.931300 6302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:39.931496 6302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:39.931816 6302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:39.931850 6302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:39.931874 6302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:39.931921 6302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:39.937897 6302 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37165
I20260924 00:19:39.937935 6430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37165 every 8 connection(s)
I20260924 00:19:39.938323 6302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 00:19:39 dist-test-slave-777x krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:39.948422 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 6302
I20260924 00:19:39.948535 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:19:39.948689 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:39.939049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53403 (local address 127.0.98.62:44063)
0924 00:19:39.939095 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:39.939097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:39.939362 (+ 265us) server_negotiation.cc:407] Connection header received
0924 00:19:39.940413 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:39.940414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:39.940427 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:39.940486 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:39.941147 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:39.941647 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:39.942663 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:39.942865 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:39.942951 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:39.945408 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:39.945410 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:39.945410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:39.945419 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:39.946869 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:39.947353 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:39.947355 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:39.947356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:39.947422 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:39.947762 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:39.947763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:39.947763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:39.947893 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:39.947978 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:39.948483 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 00:19:39.948608 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 00:19:39.948746 392 external_mini_cluster.cc:1428] Setting key 42622ade4e1d29fbc418dd49c00fddcd
I20260924 00:19:39.948909 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:39.939056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44063 (local address 127.0.98.3:53403)
0924 00:19:39.939235 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:39.939272 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:39.940303 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:39.940579 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:39.940583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:39.940857 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:39.941045 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:39.941049 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:39.941742 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:39.941744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:39.942556 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:39.942557 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:39.942958 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:39.942959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:39.943091 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:39.943589 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:39.943597 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:39.945282 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:39.946984 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:39.946991 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:39.946993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:39.947251 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:39.947532 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:39.947533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:39.947534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:39.947658 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:39.947983 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:39.947989 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:39.948148 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 00:19:39.948437 (+ 289us) client_negotiation.cc:240] Negotiation successful
0924 00:19:39.948647 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260924 00:19:39.949546 6431 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44063
I20260924 00:19:39.949669 6431 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:39.949878 6431 heartbeater.cc:511] Master 127.0.98.62:44063 requested a full tablet report, sending...
I20260924 00:19:39.950371 5972 ts_manager.cc:194] Registered new tserver with Master: a86d0976e99943f7871b599bfdcbcaf2 (127.0.98.3:37165)
I20260924 00:19:39.950783 5972 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53403
I20260924 00:19:39.961225 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/test-admin.keytab.
Sep 24 00:19:39 dist-test-slave-777x krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:39.977576 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 00:19:39.979357 6438 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:39.978514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37992 (local address 127.0.98.1:42651)
0924 00:19:39.978688 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:39.978698 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:39.978709 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:19:39.978740 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:39.978742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:39.978756 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:39.978836 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:39.979249 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37992: BlockingRecv error: recv got EOF from 127.0.0.1:37992 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
W20260924 00:19:39.979527 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 00:19:39 dist-test-slave-777x krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:39.980767 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 00:19:39 dist-test-slave-777x krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:39.981734 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 00:19:39 dist-test-slave-777x krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:19:39.993896 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:39.982264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37996 (local address 127.0.98.1:42651)
0924 00:19:39.982351 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:39.982353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:39.982433 (+ 80us) server_negotiation.cc:407] Connection header received
0924 00:19:39.982482 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:39.982483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:39.982493 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:39.982543 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:39.982929 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:39.983696 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:39.984508 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:39.984833 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:39.984889 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:39.990896 (+ 6007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:39.990907 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:39.990907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:39.990915 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:39.992679 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:39.993040 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:39.993044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:39.993044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:39.993100 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:39.993281 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:39.993282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:39.993282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:39.993423 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:39.993538 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:39.993669 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 00:19:39.993735 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 00:19:39.995782 6438 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:39.995249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38006 (local address 127.0.98.1:42651)
0924 00:19:39.995326 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:39.995328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:39.995349 (+ 21us) server_negotiation.cc:407] Connection header received
0924 00:19:39.995393 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:39.995394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:39.995404 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:39.995449 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:39.995738 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38006: BlockingRecv error: recv got EOF from 127.0.0.1:38006 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 00:19:39.995890 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 00:19:39 dist-test-slave-777x krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:39.996963 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 00:19:39 dist-test-slave-777x krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:39.997884 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 00:19:39 dist-test-slave-777x krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:19:40.005263 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:39.998443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38008 (local address 127.0.98.1:42651)
0924 00:19:39.998509 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:39.998510 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:39.998570 (+ 60us) server_negotiation.cc:407] Connection header received
0924 00:19:39.998608 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:39.998608 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:39.998619 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:39.998667 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:39.999067 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:39.999600 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:40.000352 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:40.000542 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:40.000596 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:40.002794 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:40.002796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:40.002796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:40.002802 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:40.004140 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:40.004594 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:40.004595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:40.004596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:40.004645 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:40.004825 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:40.004826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:40.004826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:40.004919 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:40.004997 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:40.005077 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 00:19:40.005133 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 00:19:40.006899 6438 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:40.006350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38016 (local address 127.0.98.1:42651)
0924 00:19:40.006383 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:40.006385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:40.006452 (+ 67us) server_negotiation.cc:407] Connection header received
0924 00:19:40.006512 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:40.006512 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:40.006520 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:40.006566 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:40.006846 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38016: BlockingRecv error: recv got EOF from 127.0.0.1:38016 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 00:19:40.007017 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790209154318147-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 00:19:40 dist-test-slave-777x krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:40.008101 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 00:19:40 dist-test-slave-777x krb5kdc[5927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:40.009057 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 00:19:40 dist-test-slave-777x krb5kdc[5927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:19:40.016248 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:40.009616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38022 (local address 127.0.98.1:42651)
0924 00:19:40.009667 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:40.009668 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:40.009680 (+ 12us) server_negotiation.cc:407] Connection header received
0924 00:19:40.009748 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:40.009748 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:40.009756 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:40.009805 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:40.010158 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:40.010602 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:40.011250 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:40.011449 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:40.011498 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:40.013639 (+ 2141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:40.013641 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:40.013641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:40.013647 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:40.015052 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:40.015461 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:40.015462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:40.015462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:40.015511 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:40.015683 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:40.015684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:40.015684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:40.015793 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:40.015875 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:40.016066 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 00:19:40.016128 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 00:19:40.017129 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 6034
I20260924 00:19:40.022642 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 6168
I20260924 00:19:40.027022 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 6302
I20260924 00:19:40.031677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 5940
2026-09-24T00:19:40Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1978 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:19:40 dist-test-slave-777x krb5kdc[6451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:19:40 dist-test-slave-777x krb5kdc[6451](info): set up 2 sockets
Sep 24 00:19:40 dist-test-slave-777x krb5kdc[6451](info): commencing operation
krb5kdc: starting...
W20260924 00:19:41.292665 392 mini_kdc.cc:121] Time spent starting KDC: real 1.241s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:19:41 dist-test-slave-777x krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:19:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:19:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 00:19:41.475786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35493
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:41.557904 6464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:41.558162 6464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:41.558200 6464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:41.559392 6464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:19:41.559464 6464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:41.559487 6464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:41.559509 6464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:19:41.559538 6464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:19:41.561180 6464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:41.561473 6464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:41.561739 6464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:41.568102 6470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:41.568109 6469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:41.568408 6464 server_base.cc:1034] running on GCE node
W20260924 00:19:41.568102 6472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:41.568849 6464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:41.569173 6464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:41.570331 6464 hybrid_clock.cc:648] HybridClock initialized: now 1790209181570315 us; error 30 us; skew 500 ppm
Sep 24 00:19:41 dist-test-slave-777x krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209181, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:41.573078 6464 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 00:19:41.573583 6464 webserver.cc:460] Webserver started at http://127.0.98.62:39405/ using document root <none> and password file <none>
I20260924 00:19:41.573818 6464 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:41.573899 6464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:41.574047 6464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:41.575169 6464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8"
format_stamp: "Formatted at 2026-09-24 00:19:41 on dist-test-slave-777x"
server_key: "779c11b2237d15f70b8bdd52dd0539b7"
server_key_iv: "743a0e4233c681290ed4a3879e36004f"
server_key_version: "encryptionkey@0"
I20260924 00:19:41.575546 6464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8"
format_stamp: "Formatted at 2026-09-24 00:19:41 on dist-test-slave-777x"
server_key: "779c11b2237d15f70b8bdd52dd0539b7"
server_key_iv: "743a0e4233c681290ed4a3879e36004f"
server_key_version: "encryptionkey@0"
I20260924 00:19:41.577095 6464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:19:41.578030 6479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:41.578284 6464 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:41.578425 6464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8"
format_stamp: "Formatted at 2026-09-24 00:19:41 on dist-test-slave-777x"
server_key: "779c11b2237d15f70b8bdd52dd0539b7"
server_key_iv: "743a0e4233c681290ed4a3879e36004f"
server_key_version: "encryptionkey@0"
I20260924 00:19:41.578517 6464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:41.621969 6464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:41.624526 6464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:41.624756 6464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:41.628813 6464 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35493
I20260924 00:19:41.628847 6531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35493 every 8 connection(s)
I20260924 00:19:41.629220 6464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:19:41.630251 6532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:41.631582 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 6464
I20260924 00:19:41.631681 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:19:41.631933 392 external_mini_cluster.cc:1428] Setting key 5db63b9809573fdd21a1f778f72f139d
I20260924 00:19:41.633008 6532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8: Bootstrap starting.
I20260924 00:19:41.633627 6532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:41.633914 6532 log.cc:824] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:41.634624 6532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8: No bootstrap required, opened a new log
I20260924 00:19:41.636693 6532 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } }
I20260924 00:19:41.636860 6532 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:41.636905 6532 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4ff0dfd51b04ab9ba3e94ad0682d7e8, State: Initialized, Role: FOLLOWER
I20260924 00:19:41.637077 6532 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } }
I20260924 00:19:41.637174 6532 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:19:41.637219 6532 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:19:41.637269 6532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:41.637820 6532 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } }
I20260924 00:19:41.637938 6532 leader_election.cc:302] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4ff0dfd51b04ab9ba3e94ad0682d7e8; no voters:
I20260924 00:19:41.638157 6532 leader_election.cc:288] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:19:41.638227 6537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:41.638434 6532 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:19:41.638433 6537 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [term 1 LEADER]: Becoming Leader. State: Replica: c4ff0dfd51b04ab9ba3e94ad0682d7e8, State: Running, Role: LEADER
I20260924 00:19:41.638656 6537 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } }
Sep 24 00:19:41 dist-test-slave-777x krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:19:41.639001 6537 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4ff0dfd51b04ab9ba3e94ad0682d7e8. Latest consensus state: current_term: 1 leader_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } }
I20260924 00:19:41.639060 6537 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:41.639115 6539 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } }
I20260924 00:19:41.639204 6539 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:41.639370 6544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:19:41.639915 6544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:19:41.641645 6544 catalog_manager.cc:1347] Generated new cluster ID: 89aba9b7ce214e8d8251917185415706
I20260924 00:19:41.641726 6544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:19:41.643051 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:41.632516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49120 (local address 127.0.98.62:35493)
0924 00:19:41.632710 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:41.632720 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:41.632759 (+ 39us) server_negotiation.cc:407] Connection header received
0924 00:19:41.633177 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:41.633186 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:41.633249 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:41.633412 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:41.634124 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:41.635144 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:41.635926 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:41.636403 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:41.636531 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:41.639354 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:41.639359 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:41.639359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:41.639375 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:41.641643 (+ 2268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:41.642047 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:41.642050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:41.642050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:41.642130 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:41.642326 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:41.642327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:41.642327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:41.642558 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:41.642656 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:41.642806 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 00:19:41.642908 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260924 00:19:41.647588 6544 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:19:41.648311 6544 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:19:41.655064 6544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8: Generated new TSK 0
I20260924 00:19:41.655261 6544 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 00:19:41.683670 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35493
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:41.766326 6558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:41.766551 6558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:41.766577 6558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:41.767810 6558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:41.767860 6558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:41.767889 6558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:19:41.769505 6558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:41.769759 6558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:41.770033 6558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:19:41.776378 6558 server_base.cc:1034] running on GCE node
W20260924 00:19:41.776484 6564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:41.776353 6566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:41.776387 6563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:41.776836 6558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:41.777105 6558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:41.778275 6558 hybrid_clock.cc:648] HybridClock initialized: now 1790209181778253 us; error 44 us; skew 500 ppm
Sep 24 00:19:41 dist-test-slave-777x krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209181, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:41.781067 6558 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 00:19:41.781706 6558 webserver.cc:460] Webserver started at http://127.0.98.1:33925/ using document root <none> and password file <none>
I20260924 00:19:41.781958 6558 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:41.782028 6558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:41.782186 6558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:41.783389 6558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "16d7a8fd2cad4de2b23a47286e3bf735"
format_stamp: "Formatted at 2026-09-24 00:19:41 on dist-test-slave-777x"
server_key: "12a4cdf8dd9e5780307fb914075a34bc"
server_key_iv: "16eb3e1a31ea338ad179ed12aa489f66"
server_key_version: "encryptionkey@0"
I20260924 00:19:41.783802 6558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "16d7a8fd2cad4de2b23a47286e3bf735"
format_stamp: "Formatted at 2026-09-24 00:19:41 on dist-test-slave-777x"
server_key: "12a4cdf8dd9e5780307fb914075a34bc"
server_key_iv: "16eb3e1a31ea338ad179ed12aa489f66"
server_key_version: "encryptionkey@0"
I20260924 00:19:41.785341 6558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:19:41.786374 6573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:41.786590 6558 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:41.786703 6558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "16d7a8fd2cad4de2b23a47286e3bf735"
format_stamp: "Formatted at 2026-09-24 00:19:41 on dist-test-slave-777x"
server_key: "12a4cdf8dd9e5780307fb914075a34bc"
server_key_iv: "16eb3e1a31ea338ad179ed12aa489f66"
server_key_version: "encryptionkey@0"
I20260924 00:19:41.786782 6558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:41.797091 6558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:41.799650 6558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:41.799806 6558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:41.800014 6558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:41.800417 6558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:41.800457 6558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:41.800480 6558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:41.800494 6558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:41.806548 6558 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41581
I20260924 00:19:41.806598 6686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41581 every 8 connection(s)
I20260924 00:19:41.806970 6558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:19:41.808771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 6558
I20260924 00:19:41.808872 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:19:41.809105 392 external_mini_cluster.cc:1428] Setting key 388ee7d2f7b47daa1a55933e2d701e96
Sep 24 00:19:41 dist-test-slave-777x krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209181, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 00:19:41.819278 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:41.807901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57881 (local address 127.0.98.62:35493)
0924 00:19:41.807977 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:41.807980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:41.808352 (+ 372us) server_negotiation.cc:407] Connection header received
0924 00:19:41.809058 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:41.809059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:41.809074 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:41.809154 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:41.810339 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:41.811135 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:41.812834 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:41.813058 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:41.813128 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:41.815807 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:41.815809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:41.815810 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:41.815818 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:41.817530 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:41.818046 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:41.818047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:41.818047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:41.818117 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:41.818421 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:41.818422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:41.818423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:41.818555 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:41.818634 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:41.819063 (+ 429us) server_negotiation.cc:299] Negotiation successful
0924 00:19:41.819162 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 00:19:41.819519 6689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:41.807880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.98.1:57881)
0924 00:19:41.808102 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:41.808152 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:41.808954 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:41.809308 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:41.809313 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:41.810008 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:41.810217 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:41.810223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:41.811322 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:41.811323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:41.812714 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:41.812717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:41.813163 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:41.813164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:41.813323 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:41.813904 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:41.813916 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:41.815663 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:41.817646 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:41.817656 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:41.817659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:41.817946 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:41.818200 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:41.818201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:41.818202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:41.818337 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:41.818639 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:41.818685 (+ 46us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:41.818896 (+ 211us) client_negotiation.cc:769] Sending connection context
0924 00:19:41.819041 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 00:19:41.819255 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":101,"threads_started":1}
I20260924 00:19:41.820233 6687 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35493
I20260924 00:19:41.820374 6687 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:41.820618 6687 heartbeater.cc:511] Master 127.0.98.62:35493 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260924 00:19:41.821286 6496 ts_manager.cc:194] Registered new tserver with Master: 16d7a8fd2cad4de2b23a47286e3bf735 (127.0.98.1:41581)
I20260924 00:19:41.821980 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 00:19:41.840323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35493
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:41.926812 6692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:41.927052 6692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:41.927080 6692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:41.928404 6692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:41.928452 6692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:41.928480 6692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:19:41.929955 6692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:41.930229 6692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:41.930478 6692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:41.935767 6700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:41.936002 6692 server_base.cc:1034] running on GCE node
W20260924 00:19:41.935767 6698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:41.935761 6697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:41.936374 6692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:41.936638 6692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:41.937819 6692 hybrid_clock.cc:648] HybridClock initialized: now 1790209181937814 us; error 39 us; skew 500 ppm
Sep 24 00:19:41 dist-test-slave-777x krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209181, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:41.940834 6692 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 00:19:41.941376 6692 webserver.cc:460] Webserver started at http://127.0.98.2:41539/ using document root <none> and password file <none>
I20260924 00:19:41.941620 6692 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:41.941690 6692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:41.941828 6692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:41.942986 6692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "20b755c043404eafbc6ffe99c13f08e7"
format_stamp: "Formatted at 2026-09-24 00:19:41 on dist-test-slave-777x"
server_key: "d54e9219c8ac21bfa941046281320556"
server_key_iv: "5411e59a854a15a203f048e6a417e7ba"
server_key_version: "encryptionkey@0"
I20260924 00:19:41.943387 6692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "20b755c043404eafbc6ffe99c13f08e7"
format_stamp: "Formatted at 2026-09-24 00:19:41 on dist-test-slave-777x"
server_key: "d54e9219c8ac21bfa941046281320556"
server_key_iv: "5411e59a854a15a203f048e6a417e7ba"
server_key_version: "encryptionkey@0"
I20260924 00:19:41.944939 6692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:19:41.945943 6707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:41.946134 6692 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:41.946305 6692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "20b755c043404eafbc6ffe99c13f08e7"
format_stamp: "Formatted at 2026-09-24 00:19:41 on dist-test-slave-777x"
server_key: "d54e9219c8ac21bfa941046281320556"
server_key_iv: "5411e59a854a15a203f048e6a417e7ba"
server_key_version: "encryptionkey@0"
I20260924 00:19:41.946410 6692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:41.955762 6692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:41.958148 6692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:41.958341 6692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:41.958621 6692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:41.958963 6692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:41.959023 6692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:41.959074 6692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:41.959108 6692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:41.965651 6692 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33353
I20260924 00:19:41.965734 6820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33353 every 8 connection(s)
I20260924 00:19:41.966156 6692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 00:19:41 dist-test-slave-777x krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209181, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:19:41.975723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 6692
I20260924 00:19:41.975811 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:19:41.976040 392 external_mini_cluster.cc:1428] Setting key ff64b833e2860b95836b2e48ab182f7c
I20260924 00:19:41.976210 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:41.966998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42587 (local address 127.0.98.62:35493)
0924 00:19:41.967062 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:41.967085 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:41.967277 (+ 192us) server_negotiation.cc:407] Connection header received
0924 00:19:41.967811 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:41.967812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:41.967824 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:41.967900 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:41.968954 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:41.969480 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:41.970512 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:41.970702 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:41.970789 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:41.973154 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:41.973156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:41.973156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:41.973164 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:41.974656 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:41.975138 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:41.975139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:41.975139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:41.975194 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:41.975454 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:41.975455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:41.975455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:41.975573 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:41.975646 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:41.975998 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 00:19:41.976079 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 00:19:41.976534 6823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:41.966963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.98.2:42587)
0924 00:19:41.967137 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:41.967182 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:41.967672 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:41.967961 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:41.967964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:41.968626 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:41.968832 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:41.968837 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:41.969568 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:41.969569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:41.970400 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:41.970402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:41.970807 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:41.970808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:41.970929 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:41.971522 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:41.971534 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:41.973065 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:41.974799 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:41.974806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:41.974809 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:41.975047 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:41.975264 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:41.975265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:41.975266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:41.975385 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:41.975650 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:41.975657 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:41.975835 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 00:19:41.975964 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 00:19:41.976218 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260924 00:19:41.977133 6821 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35493
I20260924 00:19:41.977269 6821 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:41.977499 6821 heartbeater.cc:511] Master 127.0.98.62:35493 requested a full tablet report, sending...
I20260924 00:19:41.977902 6496 ts_manager.cc:194] Registered new tserver with Master: 20b755c043404eafbc6ffe99c13f08e7 (127.0.98.2:33353)
I20260924 00:19:41.978405 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42587
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 00:19:42.004143 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35493
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:42.087595 6826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:42.087852 6826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:42.087893 6826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:42.089202 6826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:42.089272 6826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:42.089329 6826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:19:42.090965 6826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:42.091243 6826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:42.091580 6826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:42.096947 6834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:42.096957 6832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:42.096957 6831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:42.097203 6826 server_base.cc:1034] running on GCE node
I20260924 00:19:42.097427 6826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:42.097708 6826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:42.098884 6826 hybrid_clock.cc:648] HybridClock initialized: now 1790209182098863 us; error 48 us; skew 500 ppm
Sep 24 00:19:42 dist-test-slave-777x krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209182, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:42.101837 6826 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 00:19:42.102349 6826 webserver.cc:460] Webserver started at http://127.0.98.3:33221/ using document root <none> and password file <none>
I20260924 00:19:42.102566 6826 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:42.102610 6826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:42.102705 6826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:42.103753 6826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "d31733f022f44f95815023a3b85279af"
format_stamp: "Formatted at 2026-09-24 00:19:42 on dist-test-slave-777x"
server_key: "ba998811c915fce8323f64b837be877d"
server_key_iv: "741771c97c3d828a5bf4a303029566bc"
server_key_version: "encryptionkey@0"
I20260924 00:19:42.104079 6826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d31733f022f44f95815023a3b85279af"
format_stamp: "Formatted at 2026-09-24 00:19:42 on dist-test-slave-777x"
server_key: "ba998811c915fce8323f64b837be877d"
server_key_iv: "741771c97c3d828a5bf4a303029566bc"
server_key_version: "encryptionkey@0"
I20260924 00:19:42.105664 6826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:42.106492 6841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:42.106712 6826 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:42.106833 6826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "d31733f022f44f95815023a3b85279af"
format_stamp: "Formatted at 2026-09-24 00:19:42 on dist-test-slave-777x"
server_key: "ba998811c915fce8323f64b837be877d"
server_key_iv: "741771c97c3d828a5bf4a303029566bc"
server_key_version: "encryptionkey@0"
I20260924 00:19:42.106918 6826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:42.118716 6826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:42.121088 6826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:42.121243 6826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:42.121573 6826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:42.121909 6826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:42.121943 6826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:42.121966 6826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:42.122014 6826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:42.128347 6826 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42859
I20260924 00:19:42.128407 6954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42859 every 8 connection(s)
I20260924 00:19:42.128731 6826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:19:42.129570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 6826
I20260924 00:19:42.129725 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:19:42.130005 392 external_mini_cluster.cc:1428] Setting key 90b3a23be33fd6c218154e921d94ad57
Sep 24 00:19:42 dist-test-slave-777x krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209182, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:19:42.139410 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.129625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53893 (local address 127.0.98.62:35493)
0924 00:19:42.129680 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:42.129682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:42.129963 (+ 281us) server_negotiation.cc:407] Connection header received
0924 00:19:42.130634 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:42.130635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:42.130646 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:42.130684 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:42.131809 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.132423 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.133440 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.133638 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.133726 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.136140 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:42.136142 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:42.136142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:42.136158 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:42.137836 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.138314 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.138315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.138316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.138384 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.138684 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.138685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.138685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.138838 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:42.138918 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:42.139226 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 00:19:42.139309 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 00:19:42.139725 6957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.129562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.98.3:53893)
0924 00:19:42.129784 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:42.129836 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:42.130550 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:42.130787 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:42.130791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:42.131503 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:42.131704 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.131709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.132514 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.132515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.133336 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.133338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.133741 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.133742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.133876 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.134516 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:42.134526 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:42.136021 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:42.137961 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.137969 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.137972 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.138216 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.138458 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.138458 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.138459 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.138617 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.138909 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:42.138916 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:42.139071 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 00:19:42.139180 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 00:19:42.139477 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":149,"threads_started":1}
I20260924 00:19:42.140349 6955 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35493
I20260924 00:19:42.140471 6955 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:42.140735 6955 heartbeater.cc:511] Master 127.0.98.62:35493 requested a full tablet report, sending...
I20260924 00:19:42.141206 6496 ts_manager.cc:194] Registered new tserver with Master: d31733f022f44f95815023a3b85279af (127.0.98.3:42859)
I20260924 00:19:42.141772 6496 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53893
I20260924 00:19:42.142199 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260924 00:19:42.172413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35567
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35493,127.0.98.61:35567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:42.255472 6960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:42.255686 6960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:42.255708 6960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:42.257061 6960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:19:42.257114 6960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:42.257128 6960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:42.257141 6960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:19:42.257154 6960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:19:42.258674 6960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35493,127.0.98.61:35567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
W20260924 00:19:42.258783 6960 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:35493,127.0.98.61:35567'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 00:19:42.258963 6960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:42.259243 6960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:42.264652 6966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:42.264652 6968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:42.264655 6965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:42.264882 6960 server_base.cc:1034] running on GCE node
I20260924 00:19:42.265177 6960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:42.265589 6960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:42.266868 6960 hybrid_clock.cc:648] HybridClock initialized: now 1790209182266843 us; error 58 us; skew 500 ppm
Sep 24 00:19:42 dist-test-slave-777x krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209182, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:42.269701 6960 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 00:19:42.270206 6960 webserver.cc:460] Webserver started at http://127.0.98.61:41103/ using document root <none> and password file <none>
I20260924 00:19:42.270462 6960 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:42.270533 6960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:42.270685 6960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:42.271857 6960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/data/instance:
uuid: "f8c3655c29014520835e894e04fa2aa7"
format_stamp: "Formatted at 2026-09-24 00:19:42 on dist-test-slave-777x"
server_key: "e2e0c3e2c4faa9d3f5f50d4b3c31f49e"
server_key_iv: "e4c064d7af39cdb6a977510967966641"
server_key_version: "encryptionkey@0"
I20260924 00:19:42.272315 6960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/wal/instance:
uuid: "f8c3655c29014520835e894e04fa2aa7"
format_stamp: "Formatted at 2026-09-24 00:19:42 on dist-test-slave-777x"
server_key: "e2e0c3e2c4faa9d3f5f50d4b3c31f49e"
server_key_iv: "e4c064d7af39cdb6a977510967966641"
server_key_version: "encryptionkey@0"
I20260924 00:19:42.273790 6960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:42.274652 6975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:42.274834 6960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:19:42.274953 6960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/wal
uuid: "f8c3655c29014520835e894e04fa2aa7"
format_stamp: "Formatted at 2026-09-24 00:19:42 on dist-test-slave-777x"
server_key: "e2e0c3e2c4faa9d3f5f50d4b3c31f49e"
server_key_iv: "e4c064d7af39cdb6a977510967966641"
server_key_version: "encryptionkey@0"
I20260924 00:19:42.275043 6960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:42.282683 6960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:42.285416 6960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:42.285602 6960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:42.290001 6960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35567
I20260924 00:19:42.290050 7027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35567 every 8 connection(s)
I20260924 00:19:42.290499 6960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/data/info.pb
I20260924 00:19:42.291625 7028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:42.293332 7028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:19:42.297763 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 6960
Sep 24 00:19:42 dist-test-slave-777x krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209182, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:19:42.304155 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.294120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49126 (local address 127.0.98.62:35493)
0924 00:19:42.294196 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:42.294198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:42.294351 (+ 153us) server_negotiation.cc:407] Connection header received
0924 00:19:42.294794 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:42.294795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:42.294804 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:42.294871 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:42.295605 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.296116 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.297281 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.297507 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.297603 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.300621 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:42.300623 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:42.300624 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:42.300633 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:42.302478 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.302971 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.302973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.302973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.303038 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.303332 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.303333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.303333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.303560 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:42.303638 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:42.304003 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 00:19:42.304086 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 00:19:42.304486 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.294049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.0.1:49126)
0924 00:19:42.294212 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:42.294254 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:42.294710 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:42.294946 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:42.294950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:42.295200 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:42.295485 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.295490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.296310 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.296311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.297197 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.297199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.297627 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.297628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.297765 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.298399 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:42.298411 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:42.300518 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:42.302568 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.302575 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.302578 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.302885 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.303129 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.303130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.303130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.303251 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.303654 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:42.303659 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:42.303812 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 00:19:42.303966 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 00:19:42.304155 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
Sep 24 00:19:42 dist-test-slave-777x krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 00:19:42.305377 7028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:19:42.308060 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.298352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54650 (local address 127.0.98.61:35567)
0924 00:19:42.298519 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:42.298530 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:42.298558 (+ 28us) server_negotiation.cc:407] Connection header received
0924 00:19:42.298577 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:42.298578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:42.298595 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:42.298653 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:42.299421 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.300520 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.301592 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.302052 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.302116 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.305067 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:42.305074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:42.305074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:42.305082 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:42.306799 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.307185 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.307188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.307189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.307253 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.307471 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.307472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.307472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.307627 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:42.307824 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:42.307870 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 00:19:42.307950 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":110,"threads_started":1}
Sep 24 00:19:42 dist-test-slave-777x krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209182, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 00:19:42.312813 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.305676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35567 (local address 127.0.0.1:54656)
0924 00:19:42.305727 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:42.305738 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:42.305775 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:42.306017 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:42.306017 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:42.306172 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:42.306213 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.306214 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.306923 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.306923 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.307410 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.307411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.307842 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.307842 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.307884 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.308355 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:42.308360 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:42.309602 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:42.311844 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.311844 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.311845 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.312005 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.312254 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.312255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.312256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.312331 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.312552 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:42.312553 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:42.312591 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 00:19:42.312651 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 00:19:42.312735 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 00:19:42.312916 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.305735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54656 (local address 127.0.98.61:35567)
0924 00:19:42.305914 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:42.305915 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:42.305920 (+ 5us) server_negotiation.cc:407] Connection header received
0924 00:19:42.305934 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:42.305935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:42.305942 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:42.306015 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:42.306361 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.306827 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.307507 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.307734 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.307794 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.309708 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:42.309710 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:42.309710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:42.309718 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:42.311740 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.312077 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.312078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.312078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.312135 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.312393 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.312394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.312394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.312489 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:42.312536 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:42.312668 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 00:19:42.312779 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":161,"thread_start_us":98,"threads_started":1}
I20260924 00:19:42.314167 7028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7: Bootstrap starting.
I20260924 00:19:42.314801 7028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:42.315078 7028 log.cc:824] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:42.315994 7028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7: No bootstrap required, opened a new log
I20260924 00:19:42.317649 7028 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } }
I20260924 00:19:42.317775 7028 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:42.317800 7028 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c3655c29014520835e894e04fa2aa7, State: Initialized, Role: FOLLOWER
I20260924 00:19:42.317963 7028 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } }
I20260924 00:19:42.318295 7035 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } } }
I20260924 00:19:42.318406 7035 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:19:42.318439 7028 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 00:19:42.320712 7046 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:19:42.320842 7046 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:19:42.322041 6960 master_runner.cc:428] Detected that this master f8c3655c29014520835e894e04fa2aa7 is joining an existing cluster
I20260924 00:19:42.322114 6960 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:35567
I20260924 00:19:42.331243 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.323283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54662 (local address 127.0.98.61:35567)
0924 00:19:42.323353 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:42.323355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:42.323642 (+ 287us) server_negotiation.cc:407] Connection header received
0924 00:19:42.323786 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:42.323786 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:42.323795 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:42.323824 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:42.324407 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.325333 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.326407 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.327184 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.327229 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.328299 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:42.328300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:42.328300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:42.328307 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:42.329894 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.330308 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.330310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.330310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.330384 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.330575 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.330576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.330576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.330706 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:42.330784 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:42.330982 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 00:19:42.331134 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":52}
I20260924 00:19:42.331459 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.323242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49130 (local address 127.0.98.62:35493)
0924 00:19:42.323298 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:42.323301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:42.323607 (+ 306us) server_negotiation.cc:407] Connection header received
0924 00:19:42.323716 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:42.323717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:42.323730 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:42.323763 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:42.324261 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.325084 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.326170 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.326486 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.326525 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.328273 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:42.328275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:42.328276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:42.328285 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:42.330132 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.330523 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.330524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.330525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.330583 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.330838 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.330839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.330840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.330996 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:42.331062 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:42.331241 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 00:19:42.331343 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:19:42.331466 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.323412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35567 (local address 127.0.0.1:54662)
0924 00:19:42.323591 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:42.323600 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:42.323686 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:42.323904 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:42.323905 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:42.324232 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:42.324314 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.324316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.325443 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.325444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.326333 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.326335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.327299 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.327300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.327358 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.327805 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:42.327810 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:42.328235 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:42.330000 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.330001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.330002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.330211 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.330456 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.330457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.330457 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.330524 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.330774 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:42.330782 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:42.330857 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 00:19:42.330936 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 00:19:42.331059 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20260924 00:19:42.331787 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.323337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.0.1:49130)
0924 00:19:42.323536 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:42.323549 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:42.323636 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:42.323831 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:42.323831 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:42.324062 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:42.324143 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.324146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.325176 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.325177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.326074 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.326077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.326623 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.326623 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.326683 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.327225 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:42.327231 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:42.328126 (+ 895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:42.330235 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.330236 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.330236 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.330459 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.330659 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.330660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.330660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.330731 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.331090 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:42.331091 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:42.331127 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 00:19:42.331582 (+ 455us) client_negotiation.cc:240] Negotiation successful
0924 00:19:42.331701 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
Sep 24 00:19:42 dist-test-slave-777x krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209181, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 00:19:42.342991 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.334622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54664 (local address 127.0.98.61:35567)
0924 00:19:42.334669 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:42.334671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:42.334809 (+ 138us) server_negotiation.cc:407] Connection header received
0924 00:19:42.334900 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:42.334901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:42.334910 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:42.334961 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:42.335505 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.336032 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.337373 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.337552 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.337614 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.340086 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:42.340088 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:42.340088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:42.340103 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:42.341716 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.342148 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.342149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.342150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.342209 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.342420 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.342421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.342421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.342542 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:42.342622 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:42.342807 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 00:19:42.342889 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 00:19:42.343032 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.334535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35567 (local address 127.0.0.1:54664)
0924 00:19:42.334684 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:42.334740 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:42.334820 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:42.334968 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:42.334971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:42.335344 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:42.335414 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.335417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.336118 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.336120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.337196 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.337199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.337631 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.337632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.337721 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.338255 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:42.338261 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:42.339980 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:42.341818 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.341828 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.341830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.342054 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.342285 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.342286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.342286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.342354 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.342614 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:42.342621 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:42.342679 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 00:19:42.342783 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 00:19:42.342917 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260924 00:19:42.343779 6495 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:35567: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f8c3655c29014520835e894e04fa2aa7"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 35567
}
attrs {
promote: true
}
}
dest_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8"
cas_config_opid_index: -1
I20260924 00:19:42.344127 6495 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: true } }
I20260924 00:19:42.344659 7056 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f8c3655c29014520835e894e04fa2aa7 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: true } } }
I20260924 00:19:42.345088 7002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:42.345935 7056 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:35567
I20260924 00:19:42.346160 7002 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [term 1 FOLLOWER]: Refusing update from remote peer c4ff0dfd51b04ab9ba3e94ad0682d7e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 00:19:42.346212 6960 master.cc:537] Master@127.0.98.61:35567 shutting down...
I20260924 00:19:42.347221 7056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:42.347671 7055 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: true } } }
I20260924 00:19:42.347802 7055 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [sys.catalog]: This master's current role is: LEADER
W20260924 00:19:42.348639 6483 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 00:19:42.349192 6960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:42.349293 6960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 00:19:42.349272 6483 consensus_peers.cc:487] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 -> Peer f8c3655c29014520835e894e04fa2aa7 (127.0.98.61:35567): Couldn't send request to peer f8c3655c29014520835e894e04fa2aa7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:19:42.349347 6960 tablet_replica.cc:330] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7: stopping tablet replica
I20260924 00:19:42.352216 6960 master.cc:552] Master@127.0.98.61:35567 shutdown complete.
I20260924 00:19:42.352270 6960 master_runner.cc:305] Clearing existing system tablet
I20260924 00:19:42.352406 6960 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:42.352454 6960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:42.353219 6960 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:42.353814 7060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:42.353958 6960 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:42.354024 6960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:19:42.354044 6960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/wal
uuid: "f8c3655c29014520835e894e04fa2aa7"
format_stamp: "Formatted at 2026-09-24 00:19:42 on dist-test-slave-777x"
server_key: "e2e0c3e2c4faa9d3f5f50d4b3c31f49e"
server_key_iv: "e4c064d7af39cdb6a977510967966641"
server_key_version: "encryptionkey@0"
I20260924 00:19:42.354696 6960 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:42.356154 6960 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 00:19:42.356374 6960 log.cc:1196] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 00:19:42.356595 6960 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7: Deleting consensus metadata
I20260924 00:19:42.356738 6960 master_runner.cc:315] Copying system tablet from 127.0.98.62:35493
I20260924 00:19:42.357422 6960 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:35493
I20260924 00:19:42.364297 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.357799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.0.1:49138)
0924 00:19:42.357928 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:42.357939 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:42.358035 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:42.358175 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:42.358176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:42.358368 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:42.358418 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.358420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.359168 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.359169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.359871 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.359874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.360369 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.360370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.360441 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.360965 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:42.360970 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:42.361335 (+ 365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:42.363145 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.363146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.363146 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.363349 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.363572 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.363572 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.363573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.363632 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.363906 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:42.363907 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:42.363967 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 00:19:42.364057 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 00:19:42.364171 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260924 00:19:42.364297 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.357863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49138 (local address 127.0.98.62:35493)
0924 00:19:42.357912 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:42.357915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:42.358011 (+ 96us) server_negotiation.cc:407] Connection header received
0924 00:19:42.358093 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:42.358094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:42.358105 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:42.358167 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:42.358504 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.359095 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.359977 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.360273 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.360366 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.361462 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:42.361464 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:42.361464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:42.361472 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:42.362941 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.363432 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.363434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.363434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.363501 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.363710 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.363712 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.363712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.363839 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:42.363915 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:42.364085 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 00:19:42.364197 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:19:42.364625 6516 tablet_copy_service.cc:140] P c4ff0dfd51b04ab9ba3e94ad0682d7e8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f8c3655c29014520835e894e04fa2aa7 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:49138)
I20260924 00:19:42.364737 6516 tablet_copy_service.cc:161] P c4ff0dfd51b04ab9ba3e94ad0682d7e8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f8c3655c29014520835e894e04fa2aa7 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:49138: session id = f8c3655c29014520835e894e04fa2aa7-00000000000000000000000000000000
I20260924 00:19:42.365378 6516 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:19:42.366031 6960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 00:19:42.367035 6960 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7: tablet copy: Starting download of 0 data blocks...
I20260924 00:19:42.367229 6960 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7: tablet copy: Starting download of 1 WAL segments...
I20260924 00:19:42.368523 6960 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:19:42.369449 6516 tablet_copy_service.cc:342] P c4ff0dfd51b04ab9ba3e94ad0682d7e8: Request end of tablet copy session f8c3655c29014520835e894e04fa2aa7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:49138
I20260924 00:19:42.369518 6516 tablet_copy_service.cc:433] P c4ff0dfd51b04ab9ba3e94ad0682d7e8: ending tablet copy session f8c3655c29014520835e894e04fa2aa7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f8c3655c29014520835e894e04fa2aa7
I20260924 00:19:42.372660 6960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:42.374094 7070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:42.374166 7068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:42.374197 7067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:42.374781 6960 server_base.cc:1034] running on GCE node
I20260924 00:19:42.374910 6960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:42.375069 6960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:42.376221 6960 hybrid_clock.cc:648] HybridClock initialized: now 1790209182376196 us; error 43 us; skew 500 ppm
Sep 24 00:19:42 dist-test-slave-777x krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209182, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:42.378127 6960 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 00:19:42.378466 6960 webserver.cc:460] Webserver started at http://127.0.98.61:35101/ using document root <none> and password file <none>
I20260924 00:19:42.378609 6960 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:42.378659 6960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:42.379341 6960 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:19:42.379873 7077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:42.380023 6960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:19:42.380090 6960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/wal
uuid: "f8c3655c29014520835e894e04fa2aa7"
format_stamp: "Formatted at 2026-09-24 00:19:42 on dist-test-slave-777x"
server_key: "e2e0c3e2c4faa9d3f5f50d4b3c31f49e"
server_key_iv: "e4c064d7af39cdb6a977510967966641"
server_key_version: "encryptionkey@0"
I20260924 00:19:42.380156 6960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:42.394173 6960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:42.396102 6960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:42.399528 6960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35567
I20260924 00:19:42.399770 7129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35567 every 8 connection(s)
I20260924 00:19:42.399973 6960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-1/data/info.pb
I20260924 00:19:42.400859 7130 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:19:42.401522 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7: Bootstrap starting.
I20260924 00:19:42.404069 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:19:42.404372 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7: Bootstrap complete.
I20260924 00:19:42.404597 7130 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: true } }
I20260924 00:19:42.404695 7130 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f8c3655c29014520835e894e04fa2aa7, State: Initialized, Role: LEARNER
I20260924 00:19:42.404750 7130 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: true } }
I20260924 00:19:42.404975 7130 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:19:42.404987 7133 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: true } } }
I20260924 00:19:42.405074 7133 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [sys.catalog]: This master's current role is: LEARNER
I20260924 00:19:42.406422 7144 catalog_manager.cc:1259] Loaded cluster ID: 89aba9b7ce214e8d8251917185415706
I20260924 00:19:42.406519 7144 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7: loading cluster ID for follower catalog manager: success
I20260924 00:19:42.407892 7144 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7: acquiring CA information for follower catalog manager: success
I20260924 00:19:42.408262 7144 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 00:19:42.417757 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.409970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54680 (local address 127.0.98.61:35567)
0924 00:19:42.410120 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:42.410122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:42.410139 (+ 17us) server_negotiation.cc:407] Connection header received
0924 00:19:42.410177 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:42.410178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:42.410191 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:42.410279 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:42.411036 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.411812 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.413032 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.413304 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.413371 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.414955 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:42.414956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:42.414956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:42.414963 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:42.416526 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.416929 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.416930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.416930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.417000 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.417213 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.417213 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.417214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.417389 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:42.417470 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:42.417622 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 00:19:42.417686 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260924 00:19:42.452293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39793
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35493,127.0.98.61:35567,127.0.98.60:39793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:42.548408 7148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:42.548635 7148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:42.548659 7148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:42.549989 7148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:19:42.550037 7148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:42.550051 7148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:42.550065 7148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:19:42.550109 7148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:19:42.551782 7148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35493,127.0.98.61:35567,127.0.98.60:39793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:42.552080 7148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:42.552361 7148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:42.558063 7153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:42.558213 7154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:42.558041 7156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:42.558423 7148 server_base.cc:1034] running on GCE node
I20260924 00:19:42.558696 7148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:42.559078 7148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:42.560256 7148 hybrid_clock.cc:648] HybridClock initialized: now 1790209182560238 us; error 44 us; skew 500 ppm
Sep 24 00:19:42 dist-test-slave-777x krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209182, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:42.563218 7148 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 00:19:42.563689 7148 webserver.cc:460] Webserver started at http://127.0.98.60:38761/ using document root <none> and password file <none>
I20260924 00:19:42.563951 7148 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:42.564033 7148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:42.564220 7148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:42.565883 7148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/data/instance:
uuid: "152061622ae94a0ea8abcd158ad96c05"
format_stamp: "Formatted at 2026-09-24 00:19:42 on dist-test-slave-777x"
server_key: "921dabe127dbb8b226b44a56313a3e43"
server_key_iv: "466bf8fb17d6bf925fbb974d206cb415"
server_key_version: "encryptionkey@0"
I20260924 00:19:42.566579 7148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/wal/instance:
uuid: "152061622ae94a0ea8abcd158ad96c05"
format_stamp: "Formatted at 2026-09-24 00:19:42 on dist-test-slave-777x"
server_key: "921dabe127dbb8b226b44a56313a3e43"
server_key_iv: "466bf8fb17d6bf925fbb974d206cb415"
server_key_version: "encryptionkey@0"
I20260924 00:19:42.569036 7148 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260924 00:19:42.570343 7163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:42.570568 7148 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:42.570696 7148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/wal
uuid: "152061622ae94a0ea8abcd158ad96c05"
format_stamp: "Formatted at 2026-09-24 00:19:42 on dist-test-slave-777x"
server_key: "921dabe127dbb8b226b44a56313a3e43"
server_key_iv: "466bf8fb17d6bf925fbb974d206cb415"
server_key_version: "encryptionkey@0"
I20260924 00:19:42.570799 7148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:42.609190 7148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:42.611717 7148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:42.611908 7148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:42.616088 7148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39793
I20260924 00:19:42.616163 7215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39793 every 8 connection(s)
I20260924 00:19:42.616587 7148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/data/info.pb
I20260924 00:19:42.617574 7216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:42.618637 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 7148
I20260924 00:19:42.619764 7216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:19:42 dist-test-slave-777x krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209182, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 00:19:42 dist-test-slave-777x krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 00:19:42.630091 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.620928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49148 (local address 127.0.98.62:35493)
0924 00:19:42.620971 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:42.620974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:42.621084 (+ 110us) server_negotiation.cc:407] Connection header received
0924 00:19:42.621159 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:42.621160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:42.621174 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:42.621253 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:42.621782 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.622319 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.623254 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.623467 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.623563 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.625933 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:42.625935 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:42.625935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:42.625944 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:42.628021 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.628827 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.628828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.628829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.628900 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.629244 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.629245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.629246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.629419 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:42.629490 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:42.629909 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 00:19:42.629997 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:19:42.630410 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.620748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.0.1:49148)
0924 00:19:42.620948 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:42.620997 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:42.621087 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:42.621281 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:42.621285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:42.621567 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:42.621666 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.621671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.622439 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.622440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.623153 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.623155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.623572 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.623574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.623659 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.624277 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:42.624286 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:42.625808 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:42.628130 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.628137 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.628140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.628704 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.629016 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.629017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.629017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.629161 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.629519 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:42.629533 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:42.629733 (+ 200us) client_negotiation.cc:769] Sending connection context
0924 00:19:42.629870 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 00:19:42.630041 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":108,"threads_started":1}
I20260924 00:19:42.630896 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.619304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35498 (local address 127.0.98.60:39793)
0924 00:19:42.619487 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:42.619496 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:42.619524 (+ 28us) server_negotiation.cc:407] Connection header received
0924 00:19:42.619990 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:42.619997 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:42.620049 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:42.620253 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:42.621065 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.622026 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.622848 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.623281 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.623428 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.626425 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:42.626432 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:42.626433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:42.626440 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:42.628911 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.629437 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.629441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.629441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.629503 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.629742 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.629743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.629743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.629921 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:42.630639 (+ 718us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:42.630688 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 00:19:42.630773 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260924 00:19:42.630985 7216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:19:42 dist-test-slave-777x krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209182, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 00:19:42.638803 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.631366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54686 (local address 127.0.98.61:35567)
0924 00:19:42.631435 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:42.631438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:42.631687 (+ 249us) server_negotiation.cc:407] Connection header received
0924 00:19:42.631776 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:42.631777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:42.631792 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:42.631847 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:42.632128 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.632726 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.633519 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.633767 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.633861 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.635753 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:42.635762 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:42.635762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:42.635771 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:42.637488 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.637925 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.637926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.637927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.638003 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.638248 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.638249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.638249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.638378 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:42.638463 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:42.638619 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 00:19:42.638704 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 00:19:42.638823 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.631287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35567 (local address 127.0.0.1:54686)
0924 00:19:42.631624 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:42.631634 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:42.631700 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:42.631847 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:42.631848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:42.631995 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:42.632043 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.632045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.632827 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.632829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.633409 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.633411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.633860 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.633861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.633937 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.634415 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:42.634421 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:42.635653 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:42.637613 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.637614 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.637615 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.637806 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.638080 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.638081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.638081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.638147 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.638452 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:42.638453 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:42.638502 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 00:19:42.638582 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 00:19:42.638691 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325}
I20260924 00:19:42.639545 7216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39793 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:19:42 dist-test-slave-777x krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209182, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 00:19:42.645951 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.639856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39793 (local address 127.0.0.1:35504)
0924 00:19:42.639936 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:42.639948 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:42.640022 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:42.640159 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:42.640159 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:42.640373 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:42.640408 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.640410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.641122 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.641124 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.641629 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.641631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.641995 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.641996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.642033 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.642419 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:42.642423 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:42.643584 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:42.645033 (+ 1449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.645034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.645034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.645188 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.645417 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.645417 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.645418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.645464 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.645642 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:42.645643 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:42.645674 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 00:19:42.645762 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 00:19:42.645862 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 00:19:42.646030 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.639926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35504 (local address 127.0.98.60:39793)
0924 00:19:42.639992 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:42.639993 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:42.640003 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:19:42.640092 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:42.640092 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:42.640100 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:42.640151 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:42.640505 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.641030 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.641712 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.641931 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.641996 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.643658 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:42.643659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:42.643659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:42.643665 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:42.644960 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.645264 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.645264 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.645264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.645309 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.645503 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.645503 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.645503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.645594 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:42.645637 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:42.645793 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 00:19:42.645907 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":45}
I20260924 00:19:42.647380 7216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05: Bootstrap starting.
I20260924 00:19:42.648151 7216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:42.648481 7216 log.cc:824] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:42.649173 7216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05: No bootstrap required, opened a new log
I20260924 00:19:42.651075 7216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39793 } }
I20260924 00:19:42.651242 7216 raft_consensus.cc:374] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:42.651273 7216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152061622ae94a0ea8abcd158ad96c05, State: Initialized, Role: FOLLOWER
I20260924 00:19:42.651383 7216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39793 } }
I20260924 00:19:42.651688 7216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:19:42.651665 7221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39793 } } }
I20260924 00:19:42.651791 7221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 00:19:42.653900 7232 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:19:42.653968 7232 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:19:42.657086 7148 master_runner.cc:428] Detected that this master 152061622ae94a0ea8abcd158ad96c05 is joining an existing cluster
I20260924 00:19:42.657159 7148 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:39793
I20260924 00:19:42.666499 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.658507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.0.1:49164)
0924 00:19:42.658668 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:42.658678 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:42.658829 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:42.659032 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:42.659033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:42.659424 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:42.659503 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.659506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.660873 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.660874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.661847 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.661849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.662525 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.662526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.662588 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.663035 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:42.663039 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:42.663365 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:42.665302 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.665303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.665304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.665487 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.665796 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.665797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.665797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.665861 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.666186 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:42.666188 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:42.666245 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 00:19:42.666323 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 00:19:42.666413 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":98,"threads_started":1}
I20260924 00:19:42.666572 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.658552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49164 (local address 127.0.98.62:35493)
0924 00:19:42.658599 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:42.658601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:42.658760 (+ 159us) server_negotiation.cc:407] Connection header received
0924 00:19:42.658915 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:42.658916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:42.658928 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:42.658962 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:42.659614 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.660759 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.662079 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.662436 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.662515 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.663472 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:42.663473 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:42.663474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:42.663483 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:42.665191 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.665583 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.665585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.665586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.665671 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.665938 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.665939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.665939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.666087 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:42.666167 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:42.666377 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 00:19:42.666470 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 00:19:42 dist-test-slave-777x krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209181, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 00:19:42.682520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 6558
I20260924 00:19:42.684579 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.669502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35528 (local address 127.0.98.60:39793)
0924 00:19:42.669658 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:42.669660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:42.669666 (+ 6us) server_negotiation.cc:407] Connection header received
0924 00:19:42.669697 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:42.669698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:42.669719 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:42.669766 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:42.670187 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.670734 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.671880 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.672119 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.672348 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.674643 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:42.674645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:42.674645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:42.674653 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:42.683168 (+ 8515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.683636 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.683637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.683638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.683699 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.683913 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.683913 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.683914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.684056 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:42.684125 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:42.684359 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 00:19:42.684450 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":108,"threads_started":1}
I20260924 00:19:42.684635 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.669441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39793 (local address 127.0.0.1:35528)
0924 00:19:42.669498 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:42.669510 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:42.669554 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:42.669805 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:42.669806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:42.670031 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:42.670087 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.670089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.670820 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.670821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.671760 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.671763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.672367 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.672368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.672448 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.672962 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:42.672968 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:42.674537 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:42.683296 (+ 8759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.683298 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.683298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.683548 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.683779 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.683781 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.683781 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.683840 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.684121 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:42.684123 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:42.684223 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 00:19:42.684315 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 00:19:42.684466 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 00:19:42.685637 6495 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:39793: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "152061622ae94a0ea8abcd158ad96c05"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 39793
}
attrs {
promote: true
}
}
dest_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8"
cas_config_opid_index: 5
I20260924 00:19:42.685923 6495 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: true } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: true } }
I20260924 00:19:42.686447 7056 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 152061622ae94a0ea8abcd158ad96c05 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: true } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: true } } }
I20260924 00:19:42.687599 7056 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:39793
I20260924 00:19:42.687690 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: true } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: true } } }
I20260924 00:19:42.687775 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:42.688218 7148 master.cc:537] Master@127.0.98.60:39793 shutting down...
I20260924 00:19:42.688637 7190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:42.690002 7190 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [term 1 FOLLOWER]: Refusing update from remote peer c4ff0dfd51b04ab9ba3e94ad0682d7e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 00:19:42.690991 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.686595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35567 (local address 127.0.0.1:54694)
0924 00:19:42.686640 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:42.686654 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:42.686701 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:42.687122 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:42.687124 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:42.687126 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:42.687181 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.687183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.688325 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.688327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.689786 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.689789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.690715 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.690716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.690789 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.690822 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 00:19:42.690864 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 00:19:42.690914 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 00:19:42.691124 7055 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:42.691361 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.686791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54694 (local address 127.0.98.61:35567)
0924 00:19:42.686929 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:42.686931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:42.686940 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:19:42.686968 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:42.686968 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:42.686996 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:42.687083 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:42.687283 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.688159 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.689907 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.690607 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.691109 (+ 502us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.691202 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:42.691226 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 00:19:42.691284 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260924 00:19:42.691967 7104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [term 1 LEARNER]: Refusing update from remote peer c4ff0dfd51b04ab9ba3e94ad0682d7e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260924 00:19:42.692636 6481 consensus_peers.cc:487] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 -> Peer 152061622ae94a0ea8abcd158ad96c05 (127.0.98.60:39793): Couldn't send request to peer 152061622ae94a0ea8abcd158ad96c05. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:19:42.693034 7055 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 00:19:42.693696 7104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 152061622ae94a0ea8abcd158ad96c05 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: true } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: true } } }
I20260924 00:19:42.694303 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.658898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35567 (local address 127.0.0.1:54688)
0924 00:19:42.659299 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:42.659308 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:42.659399 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:42.659582 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:42.659582 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:42.659765 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:42.659827 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.659828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.660742 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.660743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.661583 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.661585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.662084 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.662085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.662136 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.681305 (+ 19169us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:42.681311 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:42.681915 (+ 604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:42.693003 (+ 11088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.693004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.693005 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.693278 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.693533 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.693534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.693534 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.693604 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.693922 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:42.693923 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:42.693977 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 00:19:42.694052 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 00:19:42.694169 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":60,"threads_started":1}
I20260924 00:19:42.694650 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.658563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54688 (local address 127.0.98.61:35567)
0924 00:19:42.658612 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:42.658614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:42.659361 (+ 747us) server_negotiation.cc:407] Connection header received
0924 00:19:42.659475 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:42.659476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:42.659488 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:42.659520 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:42.659915 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.660655 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.661664 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.661967 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.662007 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.682051 (+ 20044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:42.682053 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:42.682053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:42.682062 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:42.692911 (+ 10849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.693381 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.693383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.693384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.693453 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.693676 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.693677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.693677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.693824 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:42.693893 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:42.694439 (+ 546us) server_negotiation.cc:299] Negotiation successful
0924 00:19:42.694538 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:19:42.694872 7133 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4ff0dfd51b04ab9ba3e94ad0682d7e8. Latest consensus state: current_term: 1 leader_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: true } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: true } } }
I20260924 00:19:42.694955 7133 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [sys.catalog]: This master's current role is: LEARNER
I20260924 00:19:42.694891 7245 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: true } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: true } } }
I20260924 00:19:42.695003 7245 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [sys.catalog]: This master's current role is: LEARNER
I20260924 00:19:42.695237 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.658744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39793 (local address 127.0.0.1:35516)
0924 00:19:42.658884 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:42.658893 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:42.658957 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:42.659148 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:42.659149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:42.659391 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:42.659478 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.659481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.660428 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.660429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.661271 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.661272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.661764 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.661764 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.661817 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.662405 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:42.662410 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:42.680595 (+ 18185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:42.694164 (+ 13569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.694164 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.694164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.694375 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.694577 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.694577 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.694578 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.694628 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.694956 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:42.694956 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:42.694996 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 00:19:42.695065 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 00:19:42.695163 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260924 00:19:42.695779 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.658776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35516 (local address 127.0.98.60:39793)
0924 00:19:42.658816 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:42.658818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:42.658937 (+ 119us) server_negotiation.cc:407] Connection header received
0924 00:19:42.659055 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:42.659057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:42.659068 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:42.659139 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:42.659555 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.660342 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.661365 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.661652 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.661688 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.680707 (+ 19019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:42.680709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:42.680709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:42.680719 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:42.694080 (+ 13361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.694457 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.694458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.694458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.694511 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.694691 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.694691 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.694692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.694842 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:42.694929 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:42.695591 (+ 662us) server_negotiation.cc:299] Negotiation successful
0924 00:19:42.695679 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 00:19:42.696436 7148 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:42.696548 7148 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:42.696616 7148 tablet_replica.cc:330] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05: stopping tablet replica
I20260924 00:19:42.697099 7055 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8: attempting to promote NON_VOTER f8c3655c29014520835e894e04fa2aa7 to VOTER
I20260924 00:19:42.697126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41581
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35493
--webserver_port=33925
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35493,127.0.98.61:35567
--tserver_master_addrs=127.0.98.62:35493,127.0.98.61:35567,127.0.98.60:39793 with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:19:42.697368 7055 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: true } }
I20260924 00:19:42.697889 7104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [term 1 LEARNER]: Refusing update from remote peer c4ff0dfd51b04ab9ba3e94ad0682d7e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 00:19:42.698068 7056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260924 00:19:42.698146 6481 consensus_peers.cc:487] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 -> Peer 152061622ae94a0ea8abcd158ad96c05 (127.0.98.60:39793): Couldn't send request to peer 152061622ae94a0ea8abcd158ad96c05. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:19:42.698901 7056 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f8c3655c29014520835e894e04fa2aa7 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: true } } }
I20260924 00:19:42.699851 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: true } } }
I20260924 00:19:42.699927 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:42.699980 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: true } } }
I20260924 00:19:42.700026 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:42.700371 7104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f8c3655c29014520835e894e04fa2aa7 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: true } } }
I20260924 00:19:42.701930 7245 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: true } } }
I20260924 00:19:42.702029 7245 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:19:42.703557 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.703375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39793 (local address 127.0.0.1:35538)
0924 00:19:42.703475 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:42.703522 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 00:19:42.712105 7148 master.cc:552] Master@127.0.98.60:39793 shutdown complete.
I20260924 00:19:42.712216 7148 master_runner.cc:305] Clearing existing system tablet
I20260924 00:19:42.712383 7148 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:42.712445 7148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:42.713274 7148 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:42.713936 7248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:42.714056 7148 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:42.714105 7148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:19:42.714128 7148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/wal
uuid: "152061622ae94a0ea8abcd158ad96c05"
format_stamp: "Formatted at 2026-09-24 00:19:42 on dist-test-slave-777x"
server_key: "921dabe127dbb8b226b44a56313a3e43"
server_key_iv: "466bf8fb17d6bf925fbb974d206cb415"
server_key_version: "encryptionkey@0"
I20260924 00:19:42.715071 7148 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:42.716697 7148 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 00:19:42.716779 7148 log.cc:1196] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 00:19:42.717047 7148 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05: Deleting consensus metadata
I20260924 00:19:42.717175 7148 master_runner.cc:315] Copying system tablet from 127.0.98.62:35493
I20260924 00:19:42.717806 7148 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:35493
I20260924 00:19:42.725925 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.718228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49168 (local address 127.0.98.62:35493)
0924 00:19:42.718285 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:42.718288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:42.718365 (+ 77us) server_negotiation.cc:407] Connection header received
0924 00:19:42.718441 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:42.718442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:42.718453 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:42.718535 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:42.718974 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.719822 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.721164 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.721479 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.721560 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.722800 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:42.722802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:42.722802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:42.722811 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:42.724626 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.725060 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.725062 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.725062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.725129 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.725353 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.725354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.725355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.725501 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:42.725572 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:42.725743 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 00:19:42.725826 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 00:19:42.725951 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.718144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.0.1:49168)
0924 00:19:42.718294 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:42.718306 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:42.718381 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:42.718553 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:42.718554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:42.718793 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:42.718878 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.718880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.719929 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.719930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.721059 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.721062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.721570 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.721571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.721645 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.722218 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:42.722224 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:42.722695 (+ 471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:42.724723 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.724724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.724724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.724974 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.725206 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.725207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.725208 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.725282 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.725572 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:42.725573 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:42.725628 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 00:19:42.725706 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 00:19:42.725824 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
I20260924 00:19:42.726522 6516 tablet_copy_service.cc:140] P c4ff0dfd51b04ab9ba3e94ad0682d7e8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 152061622ae94a0ea8abcd158ad96c05 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:49168)
I20260924 00:19:42.726591 6516 tablet_copy_service.cc:161] P c4ff0dfd51b04ab9ba3e94ad0682d7e8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 152061622ae94a0ea8abcd158ad96c05 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:49168: session id = 152061622ae94a0ea8abcd158ad96c05-00000000000000000000000000000000
I20260924 00:19:42.727203 6516 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:19:42.727737 7148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 00:19:42.729254 7148 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05: tablet copy: Starting download of 0 data blocks...
I20260924 00:19:42.729472 7148 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05: tablet copy: Starting download of 1 WAL segments...
I20260924 00:19:42.730436 7148 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:19:42.731456 6516 tablet_copy_service.cc:342] P c4ff0dfd51b04ab9ba3e94ad0682d7e8: Request end of tablet copy session 152061622ae94a0ea8abcd158ad96c05-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:49168
I20260924 00:19:42.731519 6516 tablet_copy_service.cc:433] P c4ff0dfd51b04ab9ba3e94ad0682d7e8: ending tablet copy session 152061622ae94a0ea8abcd158ad96c05-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 152061622ae94a0ea8abcd158ad96c05
I20260924 00:19:42.736784 7148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:42.738485 7258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:42.738555 7255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:42.740356 7148 server_base.cc:1034] running on GCE node
W20260924 00:19:42.740448 7256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:42.740614 7148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:42.740782 7148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:42.741914 7148 hybrid_clock.cc:648] HybridClock initialized: now 1790209182741906 us; error 30 us; skew 500 ppm
Sep 24 00:19:42 dist-test-slave-777x krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209182, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:42.743468 7148 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 00:19:42.743757 7148 webserver.cc:460] Webserver started at http://127.0.98.60:39635/ using document root <none> and password file <none>
I20260924 00:19:42.743873 7148 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:42.743907 7148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:42.744743 7148 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:42.745366 7265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:42.745527 7148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:19:42.745582 7148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/wal
uuid: "152061622ae94a0ea8abcd158ad96c05"
format_stamp: "Formatted at 2026-09-24 00:19:42 on dist-test-slave-777x"
server_key: "921dabe127dbb8b226b44a56313a3e43"
server_key_iv: "466bf8fb17d6bf925fbb974d206cb415"
server_key_version: "encryptionkey@0"
I20260924 00:19:42.745640 7148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:42.762465 7148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:42.764559 7148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:42.767792 7148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39793
I20260924 00:19:42.767968 7317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39793 every 8 connection(s)
I20260924 00:19:42.768374 7148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/master-2/data/info.pb
I20260924 00:19:42.769459 7318 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:19:42.770035 7318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05: Bootstrap starting.
I20260924 00:19:42.773316 7318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:19:42.773562 7318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05: Bootstrap complete.
I20260924 00:19:42.773751 7318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: true } }
I20260924 00:19:42.773846 7318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 152061622ae94a0ea8abcd158ad96c05, State: Initialized, Role: LEARNER
I20260924 00:19:42.773892 7318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: true } }
I20260924 00:19:42.774076 7318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:19:42.774083 7321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: true } } }
I20260924 00:19:42.774343 7321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [sys.catalog]: This master's current role is: LEARNER
I20260924 00:19:42.776150 7332 catalog_manager.cc:1259] Loaded cluster ID: 89aba9b7ce214e8d8251917185415706
I20260924 00:19:42.776279 7332 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05: loading cluster ID for follower catalog manager: success
I20260924 00:19:42.777469 7332 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05: acquiring CA information for follower catalog manager: success
I20260924 00:19:42.777861 7332 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 00:19:42.808241 7246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:42.808501 7246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:42.808531 7246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:42.809957 7246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:42.810009 7246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:42.810041 7246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:19:42.811781 7246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41581
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33925
--tserver_master_addrs=127.0.98.62:35493,127.0.98.61:35567,127.0.98.60:39793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:42.812063 7246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:42.812381 7246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:42.817794 7340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:42.817795 7338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:42.817768 7337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:42.818200 7246 server_base.cc:1034] running on GCE node
I20260924 00:19:42.818403 7246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:42.818678 7246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:42.819860 7246 hybrid_clock.cc:648] HybridClock initialized: now 1790209182819846 us; error 35 us; skew 500 ppm
Sep 24 00:19:42 dist-test-slave-777x krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209182, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:42.822669 7246 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 00:19:42.823171 7246 webserver.cc:460] Webserver started at http://127.0.98.1:33925/ using document root <none> and password file <none>
I20260924 00:19:42.823459 7246 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:42.823531 7246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:42.825240 7246 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:42.825971 7347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:42.826185 7246 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:42.826701 7246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "16d7a8fd2cad4de2b23a47286e3bf735"
format_stamp: "Formatted at 2026-09-24 00:19:41 on dist-test-slave-777x"
server_key: "12a4cdf8dd9e5780307fb914075a34bc"
server_key_iv: "16eb3e1a31ea338ad179ed12aa489f66"
server_key_version: "encryptionkey@0"
I20260924 00:19:42.827170 7246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:42.838780 7246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:42.841183 7246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:42.841320 7246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:42.841526 7246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:42.841795 7246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:42.841835 7246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:42.841859 7246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:42.841873 7246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:42.848997 7246 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41581
I20260924 00:19:42.849067 7460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41581 every 8 connection(s)
I20260924 00:19:42.849390 7246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:19:42.854511 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 7246
I20260924 00:19:42.854678 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 6692
Sep 24 00:19:42 dist-test-slave-777x krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209182, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 00:19:42 dist-test-slave-777x krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209182, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 00:19:42.861212 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.850247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38301 (local address 127.0.98.61:35567)
0924 00:19:42.850402 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:42.850405 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:42.850672 (+ 267us) server_negotiation.cc:407] Connection header received
0924 00:19:42.851736 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:42.851737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:42.851751 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:42.851782 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:42.852672 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.853410 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.854720 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.855081 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.855124 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.857894 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:42.857895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:42.857896 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:42.857904 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:42.859436 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.859996 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.859997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.859998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.860057 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.860365 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.860366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.860367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.860540 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:42.860615 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:42.860968 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 00:19:42.861059 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260924 00:19:42.861478 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.850336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40095 (local address 127.0.98.60:39793)
0924 00:19:42.850478 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:42.850480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:42.850820 (+ 340us) server_negotiation.cc:407] Connection header received
0924 00:19:42.851770 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:42.851772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:42.851784 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:42.851839 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:42.852637 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.853295 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.854681 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.855693 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.855759 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.858469 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:42.858471 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:42.858471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:42.858479 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:42.860128 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.860658 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.860660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.860660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.860720 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.860933 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.860933 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.860934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.861050 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:42.861125 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:42.861269 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 00:19:42.861380 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":56,"threads_started":1}
I20260924 00:19:42.861891 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.850151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35567 (local address 127.0.98.1:38301)
0924 00:19:42.850476 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:42.850544 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:42.851619 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:42.851886 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:42.851887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:42.852304 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:42.852520 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.852527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.853523 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.853523 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.854537 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.854540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.855378 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.855379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.855520 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.856097 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:42.856107 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:42.857784 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:42.859517 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.859524 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.859527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.859901 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.860126 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.860126 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.860127 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.860291 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.860618 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:42.860624 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:42.860796 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 00:19:42.861350 (+ 554us) client_negotiation.cc:240] Negotiation successful
0924 00:19:42.861592 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":86,"thread_start_us":104,"threads_started":1}
I20260924 00:19:42.861922 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.850153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39793 (local address 127.0.98.1:40095)
0924 00:19:42.850740 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:42.850757 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:42.851619 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:42.851895 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:42.851895 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:42.852295 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:42.852520 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.852527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.853414 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.853415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.854583 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.854585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.855823 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.855824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.855897 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.856339 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:42.856345 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:42.858065 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:42.860341 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.860342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.860343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.860582 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.860796 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.860797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.860797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.860860 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.861118 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:42.861119 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:42.861167 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 00:19:42.861662 (+ 495us) client_negotiation.cc:240] Negotiation successful
0924 00:19:42.861799 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":83,"thread_start_us":98,"threads_started":1}
I20260924 00:19:42.862325 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33353
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35493
--webserver_port=41539
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35493,127.0.98.61:35567
--tserver_master_addrs=127.0.98.62:35493,127.0.98.61:35567,127.0.98.60:39793 with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:19:42.862421 7463 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35567
I20260924 00:19:42.862515 7463 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:42.862712 7463 heartbeater.cc:511] Master 127.0.98.61:35567 requested a full tablet report, sending...
I20260924 00:19:42.862939 7461 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39793
I20260924 00:19:42.862993 7461 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:42.863058 7461 heartbeater.cc:511] Master 127.0.98.60:39793 requested a full tablet report, sending...
I20260924 00:19:42.863241 7094 ts_manager.cc:194] Registered new tserver with Master: 16d7a8fd2cad4de2b23a47286e3bf735 (127.0.98.1:41581)
I20260924 00:19:42.863400 7282 ts_manager.cc:194] Registered new tserver with Master: 16d7a8fd2cad4de2b23a47286e3bf735 (127.0.98.1:41581)
Sep 24 00:19:42 dist-test-slave-777x krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209182, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:19:42.866631 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.850194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47281 (local address 127.0.98.62:35493)
0924 00:19:42.850402 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:42.850404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:42.850648 (+ 244us) server_negotiation.cc:407] Connection header received
0924 00:19:42.851707 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:42.851708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:42.851721 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:42.851784 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:42.852627 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.853421 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.854688 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.854917 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.854955 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.863856 (+ 8901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:42.863858 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:42.863858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:42.863865 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:42.865341 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.865742 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.865743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.865743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.865794 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:42.866043 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:42.866045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:42.866045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:42.866206 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:42.866292 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:42.866478 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 00:19:42.866563 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260924 00:19:42.866729 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:42.850152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.98.1:47281)
0924 00:19:42.850543 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:42.850554 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:42.851582 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:42.851868 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:42.851872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:42.852306 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:42.852519 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.852527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.853489 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.853489 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.854583 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:42.854585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:42.861826 (+ 7241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:42.861827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:42.861903 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:42.862402 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:42.862406 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:42.863779 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:42.865431 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.865432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.865432 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.865665 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.865861 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:42.865862 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:42.865862 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:42.865933 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:42.866294 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:42.866295 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:42.866352 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 00:19:42.866438 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 00:19:42.866567 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":111,"thread_start_us":103,"threads_started":1}
I20260924 00:19:42.867127 7462 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35493
I20260924 00:19:42.867193 7462 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:42.867270 7462 heartbeater.cc:511] Master 127.0.98.62:35493 requested a full tablet report, sending...
I20260924 00:19:42.867637 6495 ts_manager.cc:194] Re-registered known tserver with Master: 16d7a8fd2cad4de2b23a47286e3bf735 (127.0.98.1:41581)
I20260924 00:19:42.868114 6495 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47281
W20260924 00:19:42.948688 7469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:42.948904 7469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:42.948928 7469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:42.950114 7469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:42.950161 7469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:42.950188 7469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:19:42.951663 7469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41539
--tserver_master_addrs=127.0.98.62:35493,127.0.98.61:35567,127.0.98.60:39793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:42.951915 7469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:42.952235 7469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:42.957729 7477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:42.957741 7475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:42.957908 7474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:42.958199 7469 server_base.cc:1034] running on GCE node
I20260924 00:19:42.958444 7469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:42.958704 7469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:42.959887 7469 hybrid_clock.cc:648] HybridClock initialized: now 1790209182959880 us; error 35 us; skew 500 ppm
Sep 24 00:19:42 dist-test-slave-777x krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209182, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:42.962536 7469 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 00:19:42.963025 7469 webserver.cc:460] Webserver started at http://127.0.98.2:41539/ using document root <none> and password file <none>
I20260924 00:19:42.963264 7469 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:42.963348 7469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:42.964872 7469 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:42.965607 7484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:42.965842 7469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:42.965953 7469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "20b755c043404eafbc6ffe99c13f08e7"
format_stamp: "Formatted at 2026-09-24 00:19:41 on dist-test-slave-777x"
server_key: "d54e9219c8ac21bfa941046281320556"
server_key_iv: "5411e59a854a15a203f048e6a417e7ba"
server_key_version: "encryptionkey@0"
I20260924 00:19:42.966400 7469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:42.992821 7469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:42.995067 7469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:42.995260 7469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:42.995569 7469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:42.995919 7469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:42.995954 7469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:42.995982 7469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:42.996030 7469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:43.002025 7469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33353
I20260924 00:19:43.002070 7597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33353 every 8 connection(s)
I20260924 00:19:43.002542 7469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:19:43.008587 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 7469
I20260924 00:19:43.008754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 6826
Sep 24 00:19:43 dist-test-slave-777x krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209182, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 00:19:43 dist-test-slave-777x krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209182, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 00:19:43 dist-test-slave-777x krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209182, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 00:19:43.014657 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.003571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36753 (local address 127.0.98.60:39793)
0924 00:19:43.003615 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:43.003617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:43.003632 (+ 15us) server_negotiation.cc:407] Connection header received
0924 00:19:43.004567 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:43.004568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:43.004579 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:43.004647 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:43.005471 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.006123 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.007582 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.007827 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.007925 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.010876 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:43.010878 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:43.010878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:43.010886 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:43.012726 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.013235 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.013236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.013237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.013302 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.013648 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.013649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.013649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.013793 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:43.013838 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:43.014439 (+ 601us) server_negotiation.cc:299] Negotiation successful
0924 00:19:43.014534 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 00:19:43.015266 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.003334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39793 (local address 127.0.98.2:36753)
0924 00:19:43.003520 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:43.003560 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:43.004317 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:43.004669 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:43.004670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:43.005127 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:43.005346 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.005352 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.006240 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.006241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.007472 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.007473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.007986 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.007987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.008050 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.008742 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:43.008748 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:43.010764 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:43.012821 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:43.012827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:43.012829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:43.013144 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:43.013396 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:43.013396 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:43.013397 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:43.013558 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:43.014084 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:43.014090 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:43.014261 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 00:19:43.014819 (+ 558us) client_negotiation.cc:240] Negotiation successful
0924 00:19:43.015001 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":121,"thread_start_us":105,"threads_started":1}
I20260924 00:19:43.015725 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.003881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59211 (local address 127.0.98.61:35567)
0924 00:19:43.003947 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:43.003949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:43.004067 (+ 118us) server_negotiation.cc:407] Connection header received
0924 00:19:43.004705 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:43.004706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:43.004719 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:43.004812 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:43.005461 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.006240 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.007555 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.007825 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.007890 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.012132 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:43.012134 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:43.012134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:43.012141 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:43.014023 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.014609 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.014610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.014611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.014666 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.014872 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.014873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.014873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.015029 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:43.015071 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:43.015519 (+ 448us) server_negotiation.cc:299] Negotiation successful
0924 00:19:43.015612 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 00:19:43.015923 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.003321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51993 (local address 127.0.98.62:35493)
0924 00:19:43.003385 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:43.003388 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:43.003644 (+ 256us) server_negotiation.cc:407] Connection header received
0924 00:19:43.004434 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:43.004435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:43.004449 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:43.004507 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:43.005461 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.006221 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.007310 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.007564 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.007625 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.011846 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:43.011848 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:43.011848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:43.011855 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:43.014101 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.014554 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.014555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.014556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.014608 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.015144 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.015146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.015146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.015274 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:43.015318 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:43.015752 (+ 434us) server_negotiation.cc:299] Negotiation successful
0924 00:19:43.015848 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 00:19:43.015908 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42859
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35493
--webserver_port=33221
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35493,127.0.98.61:35567
--tserver_master_addrs=127.0.98.62:35493,127.0.98.61:35567,127.0.98.60:39793 with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:19:43.016074 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.003783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35567 (local address 127.0.98.2:59211)
0924 00:19:43.004006 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:43.004016 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:43.004318 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:43.004897 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:43.004899 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:43.005194 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:43.005347 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.005353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.006329 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.006330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.007464 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.007467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.007921 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.007922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.007986 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.009497 (+ 1511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:43.009502 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:43.012033 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:43.014116 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:43.014117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:43.014117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:43.014526 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:43.014734 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:43.014735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:43.014735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:43.014794 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:43.015352 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:43.015353 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:43.015401 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 00:19:43.015808 (+ 407us) client_negotiation.cc:240] Negotiation successful
0924 00:19:43.015934 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":104,"thread_start_us":120,"threads_started":1}
I20260924 00:19:43.016556 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.003334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.98.2:51993)
0924 00:19:43.003514 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:43.003559 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:43.004316 (+ 757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:43.004568 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:43.004571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:43.005191 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:43.005347 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.005353 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.006328 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.006328 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.007215 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.007217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.007687 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.007688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.007854 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.008510 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:43.008521 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:43.011763 (+ 3242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:43.014183 (+ 2420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:43.014183 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:43.014184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:43.014474 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:43.014675 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:43.014676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:43.014676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:43.015059 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:43.015588 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:43.015589 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:43.015633 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 00:19:43.016387 (+ 754us) client_negotiation.cc:240] Negotiation successful
0924 00:19:43.016487 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":66,"thread_start_us":88,"threads_started":1}
I20260924 00:19:43.017470 7600 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35567
I20260924 00:19:43.017481 7599 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35493
I20260924 00:19:43.017567 7598 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39793
I20260924 00:19:43.017593 7599 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:43.017817 7599 heartbeater.cc:511] Master 127.0.98.62:35493 requested a full tablet report, sending...
I20260924 00:19:43.017593 7600 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:43.017601 7598 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:43.018071 7600 heartbeater.cc:511] Master 127.0.98.61:35567 requested a full tablet report, sending...
I20260924 00:19:43.018122 7598 heartbeater.cc:511] Master 127.0.98.60:39793 requested a full tablet report, sending...
I20260924 00:19:43.018339 6495 ts_manager.cc:194] Re-registered known tserver with Master: 20b755c043404eafbc6ffe99c13f08e7 (127.0.98.2:33353)
I20260924 00:19:43.018393 7094 ts_manager.cc:194] Registered new tserver with Master: 20b755c043404eafbc6ffe99c13f08e7 (127.0.98.2:33353)
I20260924 00:19:43.018523 7282 ts_manager.cc:194] Registered new tserver with Master: 20b755c043404eafbc6ffe99c13f08e7 (127.0.98.2:33353)
I20260924 00:19:43.018885 6495 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51993
W20260924 00:19:43.098903 7605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:43.099134 7605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:43.099160 7605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:43.100390 7605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:43.100438 7605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:43.100466 7605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:19:43.101961 7605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33221
--tserver_master_addrs=127.0.98.62:35493,127.0.98.61:35567,127.0.98.60:39793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:43.102162 7605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:43.102360 7605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:43.107869 7611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:43.107806 7610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:43.107951 7605 server_base.cc:1034] running on GCE node
W20260924 00:19:43.107806 7613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:43.108328 7605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:43.108613 7605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:43.109777 7605 hybrid_clock.cc:648] HybridClock initialized: now 1790209183109747 us; error 53 us; skew 500 ppm
Sep 24 00:19:43 dist-test-slave-777x krb5kdc[6451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209183, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:43.112558 7605 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 00:19:43.113029 7605 webserver.cc:460] Webserver started at http://127.0.98.3:33221/ using document root <none> and password file <none>
I20260924 00:19:43.113261 7605 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:43.113317 7605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:43.114712 7605 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:43.115463 7620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:43.115687 7605 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:19:43.115796 7605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "d31733f022f44f95815023a3b85279af"
format_stamp: "Formatted at 2026-09-24 00:19:42 on dist-test-slave-777x"
server_key: "ba998811c915fce8323f64b837be877d"
server_key_iv: "741771c97c3d828a5bf4a303029566bc"
server_key_version: "encryptionkey@0"
I20260924 00:19:43.116153 7605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:43.132987 7605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:43.135030 7605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:43.135206 7605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:43.135464 7605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:43.135779 7605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:43.135835 7605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:43.135870 7605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:43.135918 7605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:43.142056 7605 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42859
I20260924 00:19:43.142102 7733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42859 every 8 connection(s)
I20260924 00:19:43.142505 7605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 00:19:43 dist-test-slave-777x krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209183, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 00:19:43 dist-test-slave-777x krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209183, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 00:19:43.151432 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 7605
Sep 24 00:19:43 dist-test-slave-777x krb5kdc[6451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209183, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:19:43.155349 7145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.143385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55327 (local address 127.0.98.61:35567)
0924 00:19:43.143602 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:43.143604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:43.143772 (+ 168us) server_negotiation.cc:407] Connection header received
0924 00:19:43.144916 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:43.144917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:43.144932 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:43.144986 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:43.145915 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.146745 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.148086 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.148411 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.148477 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.152468 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:43.152469 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:43.152470 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:43.152478 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:43.154222 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.154582 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.154583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.154584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.154636 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.154836 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.154837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.154837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.154965 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:43.155003 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:43.155197 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 00:19:43.155267 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260924 00:19:43.155357 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.143305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39793 (local address 127.0.98.3:45245)
0924 00:19:43.143732 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:43.143740 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:43.144773 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:43.145116 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:43.145117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:43.145568 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:43.145818 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.145824 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.146789 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.146790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.148004 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.148007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.148463 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.148464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.148572 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.149367 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:43.149374 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:43.151192 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:43.153407 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:43.153415 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:43.153418 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:43.153769 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:43.154117 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:43.154118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:43.154119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:43.154270 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:43.154583 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:43.154590 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:43.154786 (+ 196us) client_negotiation.cc:769] Sending connection context
0924 00:19:43.154907 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 00:19:43.155097 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":145,"thread_start_us":100,"threads_started":1}
I20260924 00:19:43.156189 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.143305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35567 (local address 127.0.98.3:55327)
0924 00:19:43.143622 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:43.143675 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:43.144775 (+ 1100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:43.145087 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:43.145091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:43.145591 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:43.145817 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.145822 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.146844 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.146845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.148004 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.148007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.148528 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.148528 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.148577 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.149254 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:43.149266 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:43.152332 (+ 3066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:43.154305 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:43.154305 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:43.154306 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:43.154502 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:43.154703 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:43.154704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:43.154704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:43.154755 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:43.155048 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:43.155049 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:43.155088 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 00:19:43.155473 (+ 385us) client_negotiation.cc:240] Negotiation successful
0924 00:19:43.155598 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":174,"thread_start_us":106,"threads_started":1}
I20260924 00:19:43.156551 7736 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35567
I20260924 00:19:43.156656 7736 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:43.156716 6535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.143446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59111 (local address 127.0.98.62:35493)
0924 00:19:43.143496 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:43.143498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:43.143870 (+ 372us) server_negotiation.cc:407] Connection header received
0924 00:19:43.145150 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:43.145152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:43.145163 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:43.145239 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:43.145956 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.146711 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.147957 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.148278 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.148350 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.153611 (+ 5261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:43.153612 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:43.153612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:43.153620 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:43.155520 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.155894 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.155895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.155896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.155947 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.156125 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.156125 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.156125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.156281 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:43.156334 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:43.156535 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 00:19:43.156609 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 00:19:43.157039 7736 heartbeater.cc:511] Master 127.0.98.61:35567 requested a full tablet report, sending...
I20260924 00:19:43.157083 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.143366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45245 (local address 127.0.98.60:39793)
0924 00:19:43.143448 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:43.143451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:43.143805 (+ 354us) server_negotiation.cc:407] Connection header received
0924 00:19:43.145008 (+ 1203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:43.145009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:43.145021 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:43.145093 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:43.145932 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.146659 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.148117 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.148385 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.148466 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.151305 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:43.151307 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:43.151307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:43.151317 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:43.153324 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.153864 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.153865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.153865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.153941 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.154355 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.154356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.154357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.154495 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:43.154560 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:43.156778 (+ 2218us) server_negotiation.cc:299] Negotiation successful
0924 00:19:43.156985 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 00:19:43.157040 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.143305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.98.3:59111)
0924 00:19:43.143784 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:43.143796 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:43.144796 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:43.145241 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:43.145241 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:43.145562 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:43.145818 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.145824 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.146791 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.146791 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.147829 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.147832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.148420 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.148421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.148571 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.149289 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:43.149294 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:43.153519 (+ 4225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:43.155614 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:43.155614 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:43.155615 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:43.155815 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:43.156011 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:43.156012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:43.156012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:43.156062 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:43.156390 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:43.156390 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:43.156431 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 00:19:43.156802 (+ 371us) client_negotiation.cc:240] Negotiation successful
0924 00:19:43.156912 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":461,"thread_start_us":64,"threads_started":1}
I20260924 00:19:43.157368 7734 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39793
I20260924 00:19:43.157406 7734 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:43.157469 7734 heartbeater.cc:511] Master 127.0.98.60:39793 requested a full tablet report, sending...
I20260924 00:19:43.157511 7094 ts_manager.cc:194] Registered new tserver with Master: d31733f022f44f95815023a3b85279af (127.0.98.3:42859)
I20260924 00:19:43.157819 7282 ts_manager.cc:194] Registered new tserver with Master: d31733f022f44f95815023a3b85279af (127.0.98.3:42859)
I20260924 00:19:43.157881 7735 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35493
I20260924 00:19:43.157949 7735 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:43.158035 7735 heartbeater.cc:511] Master 127.0.98.62:35493 requested a full tablet report, sending...
I20260924 00:19:43.158260 6495 ts_manager.cc:194] Re-registered known tserver with Master: d31733f022f44f95815023a3b85279af (127.0.98.3:42859)
I20260924 00:19:43.158634 6495 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59111
I20260924 00:19:43.162755 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.152557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49184 (local address 127.0.98.62:35493)
0924 00:19:43.152801 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:43.152803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:43.153330 (+ 527us) server_negotiation.cc:407] Connection header received
0924 00:19:43.153358 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:43.153359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:43.153374 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:43.153436 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:43.154283 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.155128 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.156310 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.156614 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.156672 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.159556 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:43.159558 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:43.159559 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:43.159566 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:43.161258 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.161600 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.161602 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.161602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.161663 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.162200 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.162201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.162202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.162325 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:43.162442 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:43.162542 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 00:19:43.162626 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":80,"threads_started":1}
I20260924 00:19:43.164496 6495 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 00:19:43.166481 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.153156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54704 (local address 127.0.98.61:35567)
0924 00:19:43.153199 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:43.153201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:43.158843 (+ 5642us) server_negotiation.cc:407] Connection header received
0924 00:19:43.158957 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:43.158959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:43.158974 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:43.159045 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:43.159657 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.160360 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.161254 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.161551 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.161592 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.162978 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:43.162979 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:43.162979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:43.162987 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:43.165232 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.165620 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.165621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.165621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.165675 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.165857 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.165858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.165858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.166013 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:43.166123 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:43.166256 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 00:19:43.166347 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 00:19:43.166524 6495 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:19:43.166954 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.152977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35540 (local address 127.0.98.60:39793)
0924 00:19:43.156743 (+ 3766us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:43.156745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:43.157901 (+ 1156us) server_negotiation.cc:407] Connection header received
0924 00:19:43.158037 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:43.158038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:43.158050 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:43.158106 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:43.158738 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.160041 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.161092 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.161387 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.161445 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.162978 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:43.162979 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:43.162980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:43.162987 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:43.164956 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.165358 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.165360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.165360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.165422 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.165641 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.165642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.165642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.165771 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:43.165883 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:43.166304 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 00:19:43.166837 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3745,"thread_start_us":53,"threads_started":1}
I20260924 00:19:43.174878 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.170208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39793 (local address 127.0.0.1:35550)
0924 00:19:43.170307 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:43.170321 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:43.170381 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:43.170564 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:43.170566 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:43.170567 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:43.170615 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.170616 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.171515 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.171516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.173274 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.173276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.174606 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.174607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.174697 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.174700 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:19:43.174746 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 00:19:43.174790 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":2}
I20260924 00:19:43.175416 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.170255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35550 (local address 127.0.98.60:39793)
0924 00:19:43.170328 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:43.170330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:43.170361 (+ 31us) server_negotiation.cc:407] Connection header received
0924 00:19:43.170473 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:43.170474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:43.170494 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:43.170546 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:43.170680 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.171430 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.173701 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.174397 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.174467 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.174552 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:43.175304 (+ 752us) server_negotiation.cc:299] Negotiation successful
0924 00:19:43.175348 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 00:19:43.175725 7292 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260924 00:19:43.176056 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.171080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33353 (local address 127.0.0.1:57816)
0924 00:19:43.171301 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:43.171311 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:43.171355 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:43.172449 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:43.172449 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:43.172456 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:43.172495 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.172496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.173378 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.173379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.174735 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.174738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.175825 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.175826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.175883 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.175887 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:19:43.175933 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 00:19:43.175991 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":2,"thread_start_us":58,"threads_started":1}
I20260924 00:19:43.176245 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.170885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42859 (local address 127.0.0.1:50888)
0924 00:19:43.171234 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:43.171245 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:43.171292 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:43.172081 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:43.172082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:43.172084 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:43.172135 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.172136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.173526 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.173527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.174879 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.174882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.175983 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.175984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.176046 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.176050 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:19:43.176081 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 00:19:43.176123 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":103,"threads_started":1}
I20260924 00:19:43.176379 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.171638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57816 (local address 127.0.98.2:33353)
0924 00:19:43.172280 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:43.172285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:43.172295 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:19:43.172332 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:43.172334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:43.172348 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:43.172501 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:43.172566 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.173276 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.174828 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.175651 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.175727 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.175794 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:43.176267 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 00:19:43.176309 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":81,"threads_started":1}
I20260924 00:19:43.176719 7321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4ff0dfd51b04ab9ba3e94ad0682d7e8. Latest consensus state: current_term: 1 leader_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: true } } }
I20260924 00:19:43.176819 7321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [sys.catalog]: This master's current role is: LEARNER
I20260924 00:19:43.177063 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.171318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50888 (local address 127.0.98.3:42859)
0924 00:19:43.171467 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:43.171475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:43.171488 (+ 13us) server_negotiation.cc:407] Connection header received
0924 00:19:43.171531 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:43.171533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:43.171550 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:43.172075 (+ 525us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:43.172235 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.173425 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.174987 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.175878 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.176420 (+ 542us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.176835 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:43.176870 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 00:19:43.176923 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260924 00:19:43.177109 7532 tablet_service.cc:1467] Processing CreateTablet for tablet a7d796edffa14894b3923cf276b9eabf (DEFAULT_TABLE table=test-table [id=7cdcecba8af34eb580b70e0b6cd113a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:43.177518 7532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d796edffa14894b3923cf276b9eabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:43.177716 7668 tablet_service.cc:1467] Processing CreateTablet for tablet a7d796edffa14894b3923cf276b9eabf (DEFAULT_TABLE table=test-table [id=7cdcecba8af34eb580b70e0b6cd113a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:43.178005 7668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d796edffa14894b3923cf276b9eabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:43.181565 7763 tablet_bootstrap.cc:492] T a7d796edffa14894b3923cf276b9eabf P 20b755c043404eafbc6ffe99c13f08e7: Bootstrap starting.
I20260924 00:19:43.181674 7242 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: true } } }
I20260924 00:19:43.181802 7242 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:43.181813 7764 tablet_bootstrap.cc:492] T a7d796edffa14894b3923cf276b9eabf P d31733f022f44f95815023a3b85279af: Bootstrap starting.
I20260924 00:19:43.181994 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.171636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.0.98.1:41581)
0924 00:19:43.173464 (+ 1828us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:43.173469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:43.173478 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:19:43.173517 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:43.173519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:43.173533 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:43.173733 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:43.173791 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.174800 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.176252 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.180774 (+ 4522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.180889 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.180960 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:43.181654 (+ 694us) server_negotiation.cc:299] Negotiation successful
0924 00:19:43.181712 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1778,"thread_start_us":77,"threads_started":1}
I20260924 00:19:43.182116 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.171205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41581 (local address 127.0.0.1:47590)
0924 00:19:43.171801 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:43.171812 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:43.172077 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:43.173629 (+ 1552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:43.173629 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:43.173631 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:43.173668 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.173669 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.174884 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.174885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.175939 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.175941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.181432 (+ 5491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.181433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.181529 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.181535 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 00:19:43.181597 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 00:19:43.181643 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":52,"threads_started":1}
I20260924 00:19:43.182478 7764 tablet_bootstrap.cc:654] T a7d796edffa14894b3923cf276b9eabf P d31733f022f44f95815023a3b85279af: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:43.182667 7056 raft_consensus.cc:1050] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8: attempting to promote NON_VOTER 152061622ae94a0ea8abcd158ad96c05 to VOTER
I20260924 00:19:43.182793 7764 log.cc:824] T a7d796edffa14894b3923cf276b9eabf P d31733f022f44f95815023a3b85279af: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:43.182814 7056 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: false } }
I20260924 00:19:43.182958 7395 tablet_service.cc:1467] Processing CreateTablet for tablet a7d796edffa14894b3923cf276b9eabf (DEFAULT_TABLE table=test-table [id=7cdcecba8af34eb580b70e0b6cd113a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:43.183327 7395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d796edffa14894b3923cf276b9eabf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:43.183415 7292 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [term 1 LEARNER]: Refusing update from remote peer c4ff0dfd51b04ab9ba3e94ad0682d7e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260924 00:19:43.183447 7104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [term 1 FOLLOWER]: Refusing update from remote peer c4ff0dfd51b04ab9ba3e94ad0682d7e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260924 00:19:43.183537 7764 tablet_bootstrap.cc:492] T a7d796edffa14894b3923cf276b9eabf P d31733f022f44f95815023a3b85279af: No bootstrap required, opened a new log
I20260924 00:19:43.183593 7764 ts_tablet_manager.cc:1397] T a7d796edffa14894b3923cf276b9eabf P d31733f022f44f95815023a3b85279af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:19:43.183733 7242 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260924 00:19:43.183833 7055 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260924 00:19:43.184512 7242 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 152061622ae94a0ea8abcd158ad96c05 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: false } } }
I20260924 00:19:43.185321 7763 tablet_bootstrap.cc:654] T a7d796edffa14894b3923cf276b9eabf P 20b755c043404eafbc6ffe99c13f08e7: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:43.185489 7242 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: false } } }
I20260924 00:19:43.185575 7242 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:43.185626 7242 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: false } } }
I20260924 00:19:43.185674 7242 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:43.185683 7764 raft_consensus.cc:348] T a7d796edffa14894b3923cf276b9eabf P d31733f022f44f95815023a3b85279af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31733f022f44f95815023a3b85279af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42859 } } peers { permanent_uuid: "20b755c043404eafbc6ffe99c13f08e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41581 } }
I20260924 00:19:43.185856 7764 raft_consensus.cc:374] T a7d796edffa14894b3923cf276b9eabf P d31733f022f44f95815023a3b85279af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:43.185904 7764 raft_consensus.cc:729] T a7d796edffa14894b3923cf276b9eabf P d31733f022f44f95815023a3b85279af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d31733f022f44f95815023a3b85279af, State: Initialized, Role: FOLLOWER
I20260924 00:19:43.185925 7104 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 152061622ae94a0ea8abcd158ad96c05 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: false } } }
I20260924 00:19:43.186025 7764 consensus_queue.cc:260] T a7d796edffa14894b3923cf276b9eabf P d31733f022f44f95815023a3b85279af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31733f022f44f95815023a3b85279af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42859 } } peers { permanent_uuid: "20b755c043404eafbc6ffe99c13f08e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41581 } }
I20260924 00:19:43.186095 7763 log.cc:824] T a7d796edffa14894b3923cf276b9eabf P 20b755c043404eafbc6ffe99c13f08e7: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:43.186290 7764 ts_tablet_manager.cc:1428] T a7d796edffa14894b3923cf276b9eabf P d31733f022f44f95815023a3b85279af: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:19:43.186913 7245 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: false } } }
I20260924 00:19:43.186998 7245 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:19:43.187014 7763 tablet_bootstrap.cc:492] T a7d796edffa14894b3923cf276b9eabf P 20b755c043404eafbc6ffe99c13f08e7: No bootstrap required, opened a new log
I20260924 00:19:43.187078 7763 ts_tablet_manager.cc:1397] T a7d796edffa14894b3923cf276b9eabf P 20b755c043404eafbc6ffe99c13f08e7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260924 00:19:43.187201 7735 heartbeater.cc:503] Master 127.0.98.62:35493 was elected leader, sending a full tablet report...
I20260924 00:19:43.187724 7768 tablet_bootstrap.cc:492] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735: Bootstrap starting.
I20260924 00:19:43.188335 7768 tablet_bootstrap.cc:654] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:43.188465 7056 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: false } } }
I20260924 00:19:43.188546 7056 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:43.188668 7768 log.cc:824] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:43.188858 7292 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 152061622ae94a0ea8abcd158ad96c05 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: false } } }
I20260924 00:19:43.189872 7321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: false } } }
I20260924 00:19:43.189963 7321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:19:43.190001 7763 raft_consensus.cc:348] T a7d796edffa14894b3923cf276b9eabf P 20b755c043404eafbc6ffe99c13f08e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31733f022f44f95815023a3b85279af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42859 } } peers { permanent_uuid: "20b755c043404eafbc6ffe99c13f08e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41581 } }
I20260924 00:19:43.190179 7763 raft_consensus.cc:374] T a7d796edffa14894b3923cf276b9eabf P 20b755c043404eafbc6ffe99c13f08e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:43.190225 7763 raft_consensus.cc:729] T a7d796edffa14894b3923cf276b9eabf P 20b755c043404eafbc6ffe99c13f08e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b755c043404eafbc6ffe99c13f08e7, State: Initialized, Role: FOLLOWER
I20260924 00:19:43.190362 7763 consensus_queue.cc:260] T a7d796edffa14894b3923cf276b9eabf P 20b755c043404eafbc6ffe99c13f08e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31733f022f44f95815023a3b85279af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42859 } } peers { permanent_uuid: "20b755c043404eafbc6ffe99c13f08e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41581 } }
I20260924 00:19:43.190631 7763 ts_tablet_manager.cc:1428] T a7d796edffa14894b3923cf276b9eabf P 20b755c043404eafbc6ffe99c13f08e7: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 00:19:43.190711 7599 heartbeater.cc:503] Master 127.0.98.62:35493 was elected leader, sending a full tablet report...
I20260924 00:19:43.192056 7768 tablet_bootstrap.cc:492] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735: No bootstrap required, opened a new log
I20260924 00:19:43.192117 7768 ts_tablet_manager.cc:1397] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260924 00:19:43.193518 7768 raft_consensus.cc:348] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31733f022f44f95815023a3b85279af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42859 } } peers { permanent_uuid: "20b755c043404eafbc6ffe99c13f08e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41581 } }
I20260924 00:19:43.193653 7768 raft_consensus.cc:374] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:43.193675 7768 raft_consensus.cc:729] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d7a8fd2cad4de2b23a47286e3bf735, State: Initialized, Role: FOLLOWER
I20260924 00:19:43.193790 7768 consensus_queue.cc:260] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31733f022f44f95815023a3b85279af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42859 } } peers { permanent_uuid: "20b755c043404eafbc6ffe99c13f08e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41581 } }
I20260924 00:19:43.194017 7768 ts_tablet_manager.cc:1428] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 00:19:43.194401 7462 heartbeater.cc:503] Master 127.0.98.62:35493 was elected leader, sending a full tablet report...
I20260924 00:19:43.247855 7771 raft_consensus.cc:482] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:19:43.248045 7771 raft_consensus.cc:504] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31733f022f44f95815023a3b85279af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42859 } } peers { permanent_uuid: "20b755c043404eafbc6ffe99c13f08e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41581 } }
I20260924 00:19:43.248523 7771 leader_election.cc:288] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d31733f022f44f95815023a3b85279af (127.0.98.3:42859), 20b755c043404eafbc6ffe99c13f08e7 (127.0.98.2:33353)
I20260924 00:19:43.251422 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.248853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55163 (local address 127.0.98.3:42859)
0924 00:19:43.248934 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:43.248936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:43.248947 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:19:43.248974 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:43.248976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:43.248987 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:43.249069 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:43.249249 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.249759 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.250698 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.251060 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.251143 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.251153 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:43.251332 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 00:19:43.251366 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 00:19:43.251411 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.248706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42859 (local address 127.0.98.1:55163)
0924 00:19:43.248768 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:43.248781 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:43.248824 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:43.249090 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:43.249091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:43.249094 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:43.249170 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.249172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.249847 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.249849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.250605 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.250607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.251141 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.251142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.251246 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.251249 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:19:43.251298 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 00:19:43.251340 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 00:19:43.251493 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.248924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46789 (local address 127.0.98.2:33353)
0924 00:19:43.248978 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:43.248980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:43.248988 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:19:43.249013 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:43.249014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:43.249025 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:43.249111 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:43.249258 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.249744 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.250713 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.251092 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.251203 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.251213 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:43.251385 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 00:19:43.251424 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 00:19:43.251549 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.248842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33353 (local address 127.0.98.1:46789)
0924 00:19:43.248894 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:43.248903 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:43.248932 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:43.249118 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:43.249119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:43.249122 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:43.249181 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.249182 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.249833 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.249834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.250610 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:43.250612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.251202 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.251203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:43.251289 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.251292 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:19:43.251376 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 00:19:43.251417 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":32}
I20260924 00:19:43.251911 7552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d796edffa14894b3923cf276b9eabf" candidate_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b755c043404eafbc6ffe99c13f08e7" is_pre_election: true
I20260924 00:19:43.252086 7552 raft_consensus.cc:2454] T a7d796edffa14894b3923cf276b9eabf P 20b755c043404eafbc6ffe99c13f08e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d7a8fd2cad4de2b23a47286e3bf735 in term 0.
I20260924 00:19:43.252519 7688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d796edffa14894b3923cf276b9eabf" candidate_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d31733f022f44f95815023a3b85279af" is_pre_election: true
I20260924 00:19:43.252663 7688 raft_consensus.cc:2454] T a7d796edffa14894b3923cf276b9eabf P d31733f022f44f95815023a3b85279af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d7a8fd2cad4de2b23a47286e3bf735 in term 0.
I20260924 00:19:43.252925 7351 leader_election.cc:302] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d7a8fd2cad4de2b23a47286e3bf735, 20b755c043404eafbc6ffe99c13f08e7; no voters:
I20260924 00:19:43.253047 7771 raft_consensus.cc:2781] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:19:43.253103 7771 raft_consensus.cc:482] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:19:43.253125 7771 raft_consensus.cc:3037] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:43.253850 7771 raft_consensus.cc:504] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31733f022f44f95815023a3b85279af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42859 } } peers { permanent_uuid: "20b755c043404eafbc6ffe99c13f08e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41581 } }
I20260924 00:19:43.253983 7771 leader_election.cc:288] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735 [CANDIDATE]: Term 1 election: Requested vote from peers d31733f022f44f95815023a3b85279af (127.0.98.3:42859), 20b755c043404eafbc6ffe99c13f08e7 (127.0.98.2:33353)
I20260924 00:19:43.254137 7552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d796edffa14894b3923cf276b9eabf" candidate_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b755c043404eafbc6ffe99c13f08e7"
I20260924 00:19:43.254149 7688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d796edffa14894b3923cf276b9eabf" candidate_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d31733f022f44f95815023a3b85279af"
W20260924 00:19:43.254209 7601 tablet.cc:2369] T a7d796edffa14894b3923cf276b9eabf P 20b755c043404eafbc6ffe99c13f08e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:19:43.254233 7688 raft_consensus.cc:3037] T a7d796edffa14894b3923cf276b9eabf P d31733f022f44f95815023a3b85279af [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:43.254235 7552 raft_consensus.cc:3037] T a7d796edffa14894b3923cf276b9eabf P 20b755c043404eafbc6ffe99c13f08e7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:43.254930 7688 raft_consensus.cc:2454] T a7d796edffa14894b3923cf276b9eabf P d31733f022f44f95815023a3b85279af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d7a8fd2cad4de2b23a47286e3bf735 in term 1.
I20260924 00:19:43.255043 7552 raft_consensus.cc:2454] T a7d796edffa14894b3923cf276b9eabf P 20b755c043404eafbc6ffe99c13f08e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d7a8fd2cad4de2b23a47286e3bf735 in term 1.
I20260924 00:19:43.255095 7351 leader_election.cc:302] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d7a8fd2cad4de2b23a47286e3bf735, d31733f022f44f95815023a3b85279af; no voters:
I20260924 00:19:43.255193 7771 raft_consensus.cc:2781] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:43.255339 7771 raft_consensus.cc:686] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735 [term 1 LEADER]: Becoming Leader. State: Replica: 16d7a8fd2cad4de2b23a47286e3bf735, State: Running, Role: LEADER
I20260924 00:19:43.255414 7771 consensus_queue.cc:237] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31733f022f44f95815023a3b85279af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42859 } } peers { permanent_uuid: "20b755c043404eafbc6ffe99c13f08e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41581 } }
I20260924 00:19:43.256333 6495 catalog_manager.cc:5495] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16d7a8fd2cad4de2b23a47286e3bf735 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d31733f022f44f95815023a3b85279af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b755c043404eafbc6ffe99c13f08e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41581 } health_report { overall_health: HEALTHY } } }
I20260924 00:19:43.272281 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.269694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47604 (local address 127.0.98.1:41581)
0924 00:19:43.269745 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:43.269747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:43.269763 (+ 16us) server_negotiation.cc:407] Connection header received
0924 00:19:43.269805 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:43.269806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:43.269826 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:43.269888 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:43.270032 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.270551 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.271357 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.271557 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.271613 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.271816 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:43.271968 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:43.272039 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:43.272139 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 00:19:43.272160 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:19:43.274853 7552 raft_consensus.cc:1261] T a7d796edffa14894b3923cf276b9eabf P 20b755c043404eafbc6ffe99c13f08e7 [term 1 FOLLOWER]: Refusing update from remote peer 16d7a8fd2cad4de2b23a47286e3bf735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:43.274902 7688 raft_consensus.cc:1261] T a7d796edffa14894b3923cf276b9eabf P d31733f022f44f95815023a3b85279af [term 1 FOLLOWER]: Refusing update from remote peer 16d7a8fd2cad4de2b23a47286e3bf735: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:43.275283 7771 consensus_queue.cc:1035] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b755c043404eafbc6ffe99c13f08e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:43.275496 7772 consensus_queue.cc:1035] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d31733f022f44f95815023a3b85279af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:43.278385 7775 mvcc.cc:205] Tried to move back new op lower bound from 7332696814690590720 to 7332696814614769664. Current Snapshot: MvccSnapshot[applied={T|T < 7332696814690590720}]
I20260924 00:19:43.279268 7776 mvcc.cc:205] Tried to move back new op lower bound from 7332696814690590720 to 7332696814614769664. Current Snapshot: MvccSnapshot[applied={T|T < 7332696814690590720}]
I20260924 00:19:43.281122 6496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:43.281222 6496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:43.282552 6496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4ff0dfd51b04ab9ba3e94ad0682d7e8: Sending DeleteTablet for 3 replicas of tablet a7d796edffa14894b3923cf276b9eabf
I20260924 00:19:43.282920 7668 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d796edffa14894b3923cf276b9eabf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50888
I20260924 00:19:43.282902 7395 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d796edffa14894b3923cf276b9eabf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47590
I20260924 00:19:43.283110 7532 tablet_service.cc:1514] Processing DeleteTablet for tablet a7d796edffa14894b3923cf276b9eabf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57816
I20260924 00:19:43.283200 7786 tablet_replica.cc:330] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735: stopping tablet replica
I20260924 00:19:43.283291 7786 raft_consensus.cc:2229] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:19:43.283384 7786 raft_consensus.cc:2258] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:43.283859 7786 ts_tablet_manager.cc:1905] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:43.284372 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 6464
I20260924 00:19:43.286036 7786 ts_tablet_manager.cc:1918] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:19:43.286114 7786 log.cc:1196] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-0/wal/wals/a7d796edffa14894b3923cf276b9eabf
I20260924 00:19:43.286384 7786 ts_tablet_manager.cc:1939] T a7d796edffa14894b3923cf276b9eabf P 16d7a8fd2cad4de2b23a47286e3bf735: Deleting consensus metadata
I20260924 00:19:43.287118 7785 tablet_replica.cc:330] T a7d796edffa14894b3923cf276b9eabf P d31733f022f44f95815023a3b85279af: stopping tablet replica
I20260924 00:19:43.287235 7785 raft_consensus.cc:2229] T a7d796edffa14894b3923cf276b9eabf P d31733f022f44f95815023a3b85279af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:43.287361 7785 raft_consensus.cc:2258] T a7d796edffa14894b3923cf276b9eabf P d31733f022f44f95815023a3b85279af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:43.287549 7787 tablet_replica.cc:330] T a7d796edffa14894b3923cf276b9eabf P 20b755c043404eafbc6ffe99c13f08e7: stopping tablet replica
I20260924 00:19:43.287673 7787 raft_consensus.cc:2229] T a7d796edffa14894b3923cf276b9eabf P 20b755c043404eafbc6ffe99c13f08e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:43.287788 7787 raft_consensus.cc:2258] T a7d796edffa14894b3923cf276b9eabf P 20b755c043404eafbc6ffe99c13f08e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:43.288443 7787 ts_tablet_manager.cc:1905] T a7d796edffa14894b3923cf276b9eabf P 20b755c043404eafbc6ffe99c13f08e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:43.290223 7787 ts_tablet_manager.cc:1918] T a7d796edffa14894b3923cf276b9eabf P 20b755c043404eafbc6ffe99c13f08e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:19:43.290299 7787 log.cc:1196] T a7d796edffa14894b3923cf276b9eabf P 20b755c043404eafbc6ffe99c13f08e7: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/wal/wals/a7d796edffa14894b3923cf276b9eabf
I20260924 00:19:43.290613 7787 ts_tablet_manager.cc:1939] T a7d796edffa14894b3923cf276b9eabf P 20b755c043404eafbc6ffe99c13f08e7: Deleting consensus metadata
W20260924 00:19:43.291154 7350 connection.cc:535] server connection from 127.0.0.1:47590 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 00:19:43.291513 7785 ts_tablet_manager.cc:1905] T a7d796edffa14894b3923cf276b9eabf P d31733f022f44f95815023a3b85279af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:43.292572 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.283374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35560 (local address 127.0.98.60:39793)
0924 00:19:43.283410 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:43.283425 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:43.283452 (+ 27us) server_negotiation.cc:407] Connection header received
0924 00:19:43.283603 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:43.283605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:43.283617 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:43.283648 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:43.285194 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.285938 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.287021 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.287287 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.287354 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.289107 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:43.289109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:43.289109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:43.289118 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:43.290972 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.291612 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.291614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.291614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.291674 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.291896 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.291897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.291898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.292023 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:43.292098 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:43.292343 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 00:19:43.292429 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 00:19:43.293202 7785 ts_tablet_manager.cc:1918] T a7d796edffa14894b3923cf276b9eabf P d31733f022f44f95815023a3b85279af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:19:43.293295 7785 log.cc:1196] T a7d796edffa14894b3923cf276b9eabf P d31733f022f44f95815023a3b85279af: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/a7d796edffa14894b3923cf276b9eabf
I20260924 00:19:43.293587 7785 ts_tablet_manager.cc:1939] T a7d796edffa14894b3923cf276b9eabf P d31733f022f44f95815023a3b85279af: Deleting consensus metadata
W20260924 00:19:43.293996 7621 connection.cc:441] server connection from 127.0.0.1:50888 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50888 (request call id 1) could send its response
W20260924 00:19:43.296592 7485 connection.cc:441] server connection from 127.0.0.1:57816 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57816 (request call id 1) could send its response
I20260924 00:19:43.302742 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.294310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54714 (local address 127.0.98.61:35567)
0924 00:19:43.294359 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:43.294361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:43.294605 (+ 244us) server_negotiation.cc:407] Connection header received
0924 00:19:43.294636 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:43.294637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:43.294648 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:43.294709 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:43.295369 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.296147 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.297224 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.297448 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.297541 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.299063 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:43.299065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:43.299065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:43.299073 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:43.301200 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.301785 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.301786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.301787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.301851 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.302107 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.302108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.302108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.302254 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:43.302356 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:43.302554 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 00:19:43.302643 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:19:43.302735 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:43.294148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.0.98.60:39793)
0924 00:19:43.294186 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:43.294187 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:43.294197 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:19:43.294220 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:43.294221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:43.294230 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:43.294279 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:43.295401 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.295963 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.297145 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:43.297463 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:43.297542 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:43.299259 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:43.299261 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:43.299261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:43.299269 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:43.301032 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.301796 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.301797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.301798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.301870 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:43.302137 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:43.302138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:43.302138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:43.302265 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:43.302385 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:43.302514 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 00:19:43.302596 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:19:44.280661 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.280411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.98.3:53773)
0924 00:19:44.280540 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:44.280614 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
W20260924 00:19:44.280923 7735 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35493 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
I20260924 00:19:44.281289 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.281032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.98.2:43539)
0924 00:19:44.281170 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:44.281244 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":55,"threads_started":1}
W20260924 00:19:44.281659 7599 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35493 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
I20260924 00:19:44.283608 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.283392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.98.1:51317)
0924 00:19:44.283505 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:44.283568 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
W20260924 00:19:44.283972 7462 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35493 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
I20260924 00:19:44.783185 7798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c4ff0dfd51b04ab9ba3e94ad0682d7e8)
I20260924 00:19:44.783324 7798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: false } }
I20260924 00:19:44.783588 7799 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c4ff0dfd51b04ab9ba3e94ad0682d7e8)
I20260924 00:19:44.783655 7798 leader_election.cc:288] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c4ff0dfd51b04ab9ba3e94ad0682d7e8 (127.0.98.62:35493), f8c3655c29014520835e894e04fa2aa7 (127.0.98.61:35567)
I20260924 00:19:44.783740 7799 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: false } }
I20260924 00:19:44.784024 7799 leader_election.cc:288] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c4ff0dfd51b04ab9ba3e94ad0682d7e8 (127.0.98.62:35493), 152061622ae94a0ea8abcd158ad96c05 (127.0.98.60:39793)
I20260924 00:19:44.784332 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.784116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.0.1:49670)
0924 00:19:44.784246 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:44.784293 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
W20260924 00:19:44.784453 7080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
I20260924 00:19:44.784003 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.783791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.0.1:49660)
0924 00:19:44.783920 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:44.783956 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":39,"threads_started":1}
W20260924 00:19:44.784587 7268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
I20260924 00:19:44.785043 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.784955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.0.1:49678)
0924 00:19:44.784991 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:44.785018 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 00:19:44.785058 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.784915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.0.1:49686)
0924 00:19:44.785005 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:44.785032 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260924 00:19:44.785137 7268 leader_election.cc:334] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c4ff0dfd51b04ab9ba3e94ad0682d7e8 (127.0.98.62:35493): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
W20260924 00:19:44.785276 7080 leader_election.cc:334] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c4ff0dfd51b04ab9ba3e94ad0682d7e8 (127.0.98.62:35493): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
I20260924 00:19:44.787359 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.784008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54724 (local address 127.0.98.61:35567)
0924 00:19:44.784138 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:44.784141 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:44.784197 (+ 56us) server_negotiation.cc:407] Connection header received
0924 00:19:44.784309 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:44.784310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:44.784322 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:44.784395 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:44.784596 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:44.785370 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:44.786428 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:44.787012 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:44.787110 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:44.787124 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:44.787262 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 00:19:44.787298 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":41,"threads_started":1}
I20260924 00:19:44.787364 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.783864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35567 (local address 127.0.0.1:54724)
0924 00:19:44.784114 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:44.784128 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:44.784227 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:44.784452 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:44.784453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:44.784456 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:44.784515 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:44.784517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:44.785469 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:44.785470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:44.786336 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:44.786338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:44.787109 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:44.787110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:44.787178 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:44.787182 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:19:44.787254 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 00:19:44.787299 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":42,"threads_started":1}
I20260924 00:19:44.787624 7104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "152061622ae94a0ea8abcd158ad96c05" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f8c3655c29014520835e894e04fa2aa7" is_pre_election: true
I20260924 00:19:44.787752 7104 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 152061622ae94a0ea8abcd158ad96c05 in term 1.
I20260924 00:19:44.787813 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.784424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35572 (local address 127.0.98.60:39793)
0924 00:19:44.784686 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:44.784688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:44.784695 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:19:44.784723 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:44.784724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:44.784731 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:44.784967 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:44.785001 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:44.785772 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:44.787001 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:44.787351 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:44.787437 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:44.787450 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:44.787704 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 00:19:44.787749 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":50,"threads_started":1}
I20260924 00:19:44.787904 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.784117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39793 (local address 127.0.0.1:35572)
0924 00:19:44.784325 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:44.784335 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:44.784471 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:44.784850 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:44.784851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:44.784853 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:44.784902 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:44.784903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:44.785865 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:44.785866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:44.786910 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:44.786913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:44.787591 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:44.787591 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:44.787636 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:44.787639 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:19:44.787679 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 00:19:44.787707 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":68,"threads_started":1}
I20260924 00:19:44.787930 7269 leader_election.cc:302] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 152061622ae94a0ea8abcd158ad96c05, f8c3655c29014520835e894e04fa2aa7; no voters: c4ff0dfd51b04ab9ba3e94ad0682d7e8
I20260924 00:19:44.788126 7798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:19:44.788203 7798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c4ff0dfd51b04ab9ba3e94ad0682d7e8)
I20260924 00:19:44.788245 7798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:19:44.788208 7292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8c3655c29014520835e894e04fa2aa7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "152061622ae94a0ea8abcd158ad96c05" is_pre_election: true
I20260924 00:19:44.788964 7798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: false } }
I20260924 00:19:44.789124 7292 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8c3655c29014520835e894e04fa2aa7 in current term 2: Already voted for candidate 152061622ae94a0ea8abcd158ad96c05 in this term.
I20260924 00:19:44.789181 7798 leader_election.cc:288] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [CANDIDATE]: Term 2 election: Requested vote from peers c4ff0dfd51b04ab9ba3e94ad0682d7e8 (127.0.98.62:35493), f8c3655c29014520835e894e04fa2aa7 (127.0.98.61:35567)
I20260924 00:19:44.789242 7104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "152061622ae94a0ea8abcd158ad96c05" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f8c3655c29014520835e894e04fa2aa7"
I20260924 00:19:44.789309 7104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:19:44.789320 7079 leader_election.cc:302] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8c3655c29014520835e894e04fa2aa7; no voters: 152061622ae94a0ea8abcd158ad96c05, c4ff0dfd51b04ab9ba3e94ad0682d7e8
I20260924 00:19:44.789903 7104 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 152061622ae94a0ea8abcd158ad96c05 in term 2.
I20260924 00:19:44.789988 7799 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:19:44.790073 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.789179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.0.1:49708)
0924 00:19:44.790018 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:44.790049 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 00:19:44.790113 7269 leader_election.cc:302] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 152061622ae94a0ea8abcd158ad96c05, f8c3655c29014520835e894e04fa2aa7; no voters:
I20260924 00:19:44.790194 7798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:19:44.790329 7798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [term 2 LEADER]: Becoming Leader. State: Replica: 152061622ae94a0ea8abcd158ad96c05, State: Running, Role: LEADER
I20260924 00:19:44.790395 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.790258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.0.1:49718)
0924 00:19:44.790307 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:44.790329 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 00:19:44.790433 7268 leader_election.cc:334] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c4ff0dfd51b04ab9ba3e94ad0682d7e8 (127.0.98.62:35493): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
I20260924 00:19:44.790429 7798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: false } }
I20260924 00:19:44.790745 7808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 152061622ae94a0ea8abcd158ad96c05. Latest consensus state: current_term: 2 leader_uuid: "152061622ae94a0ea8abcd158ad96c05" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: false } } }
I20260924 00:19:44.790843 7808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:44.791002 7810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:19:44.791359 7810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:19:44.791414 7810 catalog_manager.cc:1259] Loaded cluster ID: 89aba9b7ce214e8d8251917185415706
I20260924 00:19:44.791431 7810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:19:44.791543 7810 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:19:44.791591 7810 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05: Loaded TSK: 0
I20260924 00:19:44.791683 7810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:19:44.841859 7282 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35568:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:19:44.843467 7282 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:19:44.844612 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.844457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.0.1:49740)
0924 00:19:44.844532 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:44.844572 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 00:19:44.844645 7104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [term 2 FOLLOWER]: Refusing update from remote peer 152061622ae94a0ea8abcd158ad96c05: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 00:19:44.844944 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.844846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.0.1:49748)
0924 00:19:44.844886 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:44.844921 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 00:19:44.844985 7808 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260924 00:19:44.845094 7268 consensus_peers.cc:487] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 -> Peer c4ff0dfd51b04ab9ba3e94ad0682d7e8 (127.0.98.62:35493): Couldn't send request to peer c4ff0dfd51b04ab9ba3e94ad0682d7e8. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 00:19:44.845587 7799 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 152061622ae94a0ea8abcd158ad96c05. Latest consensus state: current_term: 2 leader_uuid: "152061622ae94a0ea8abcd158ad96c05" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: false } } }
I20260924 00:19:44.845701 7799 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:19:44.846144 7808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "152061622ae94a0ea8abcd158ad96c05" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: false } } }
I20260924 00:19:44.846225 7808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05 [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:44.846321 7812 mvcc.cc:205] Tried to move back new op lower bound from 7332696821121024000 to 7332696820902252544. Current Snapshot: MvccSnapshot[applied={T|T < 7332696821121024000}]
I20260924 00:19:44.846441 7799 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "152061622ae94a0ea8abcd158ad96c05" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "c4ff0dfd51b04ab9ba3e94ad0682d7e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35493 } } peers { permanent_uuid: "f8c3655c29014520835e894e04fa2aa7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35567 } attrs { promote: false } } peers { permanent_uuid: "152061622ae94a0ea8abcd158ad96c05" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39793 } attrs { promote: false } } }
I20260924 00:19:44.846537 7799 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c3655c29014520835e894e04fa2aa7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:19:44.851037 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.847968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41581 (local address 127.0.0.1:47612)
0924 00:19:44.848017 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:44.848028 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:44.848101 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:44.848328 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:44.848329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:44.848331 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:44.848382 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:44.848383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:44.849330 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:44.849331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:44.850072 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:44.850074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:44.850807 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:44.850808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:44.850869 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:44.850874 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:19:44.850931 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 00:19:44.850970 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 00:19:44.851055 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.848029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47612 (local address 127.0.98.1:41581)
0924 00:19:44.848144 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:44.848146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:44.848155 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:19:44.848214 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:44.848215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:44.848227 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:44.848322 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:44.848466 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:44.849246 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:44.850155 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:44.850722 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:44.850798 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:44.850811 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:44.850947 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 00:19:44.850989 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":61,"threads_started":1}
I20260924 00:19:44.851507 7395 tablet_service.cc:1467] Processing CreateTablet for tablet c909ae217a6e4cf581f1e038306243c4 (DEFAULT_TABLE table=test-table [id=47c168c4ab4d4a62829e497a400dda10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:44.851640 7395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c909ae217a6e4cf581f1e038306243c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:44.852249 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.848063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33353 (local address 127.0.0.1:57820)
0924 00:19:44.848187 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:44.848197 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:44.848247 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:44.848597 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:44.848598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:44.848600 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:44.848635 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:44.848636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:44.849997 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:44.849998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:44.851187 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:44.851194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:44.851952 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:44.851953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:44.852022 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:44.852025 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:19:44.852074 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 00:19:44.852108 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":41}
I20260924 00:19:44.852306 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.848321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50904 (local address 127.0.98.3:42859)
0924 00:19:44.848433 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:44.848435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:44.848443 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:19:44.848467 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:44.848468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:44.848479 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:44.848537 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:44.848703 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:44.849257 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:44.851434 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:44.851956 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:44.852024 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:44.852038 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:44.852161 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 00:19:44.852226 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
I20260924 00:19:44.852324 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.848138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42859 (local address 127.0.0.1:50904)
0924 00:19:44.848276 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:44.848286 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:44.848326 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:44.848582 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:44.848583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:44.848585 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:44.848627 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:44.848628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:44.850161 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:44.850162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:44.851335 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:44.851338 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:44.852027 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:44.852028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:44.852084 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:44.852088 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:19:44.852141 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 00:19:44.852204 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260924 00:19:44.852845 7668 tablet_service.cc:1467] Processing CreateTablet for tablet c909ae217a6e4cf581f1e038306243c4 (DEFAULT_TABLE table=test-table [id=47c168c4ab4d4a62829e497a400dda10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:44.852957 7668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c909ae217a6e4cf581f1e038306243c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:44.853075 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.848151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57820 (local address 127.0.98.2:33353)
0924 00:19:44.848465 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:44.848466 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:44.848474 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:19:44.848499 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:44.848500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:44.848510 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:44.848693 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:44.848731 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:44.849878 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:44.851288 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:44.851867 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:44.851942 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:44.851954 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:44.852867 (+ 913us) server_negotiation.cc:299] Negotiation successful
0924 00:19:44.852993 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":69,"threads_started":1}
I20260924 00:19:44.853282 7532 tablet_service.cc:1467] Processing CreateTablet for tablet c909ae217a6e4cf581f1e038306243c4 (DEFAULT_TABLE table=test-table [id=47c168c4ab4d4a62829e497a400dda10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:44.853427 7532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c909ae217a6e4cf581f1e038306243c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:44.854182 7821 tablet_bootstrap.cc:492] T c909ae217a6e4cf581f1e038306243c4 P d31733f022f44f95815023a3b85279af: Bootstrap starting.
I20260924 00:19:44.854593 7821 tablet_bootstrap.cc:654] T c909ae217a6e4cf581f1e038306243c4 P d31733f022f44f95815023a3b85279af: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:44.854734 7822 tablet_bootstrap.cc:492] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7: Bootstrap starting.
I20260924 00:19:44.854838 7820 tablet_bootstrap.cc:492] T c909ae217a6e4cf581f1e038306243c4 P 16d7a8fd2cad4de2b23a47286e3bf735: Bootstrap starting.
I20260924 00:19:44.855095 7822 tablet_bootstrap.cc:654] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:44.855144 7820 tablet_bootstrap.cc:654] T c909ae217a6e4cf581f1e038306243c4 P 16d7a8fd2cad4de2b23a47286e3bf735: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:44.855669 7820 tablet_bootstrap.cc:492] T c909ae217a6e4cf581f1e038306243c4 P 16d7a8fd2cad4de2b23a47286e3bf735: No bootstrap required, opened a new log
I20260924 00:19:44.855726 7820 ts_tablet_manager.cc:1397] T c909ae217a6e4cf581f1e038306243c4 P 16d7a8fd2cad4de2b23a47286e3bf735: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:44.855739 7822 tablet_bootstrap.cc:492] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7: No bootstrap required, opened a new log
I20260924 00:19:44.855789 7822 ts_tablet_manager.cc:1397] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:44.855844 7820 raft_consensus.cc:348] T c909ae217a6e4cf581f1e038306243c4 P 16d7a8fd2cad4de2b23a47286e3bf735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41581 } } peers { permanent_uuid: "20b755c043404eafbc6ffe99c13f08e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "d31733f022f44f95815023a3b85279af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42859 } }
I20260924 00:19:44.855911 7820 raft_consensus.cc:374] T c909ae217a6e4cf581f1e038306243c4 P 16d7a8fd2cad4de2b23a47286e3bf735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:44.855926 7820 raft_consensus.cc:729] T c909ae217a6e4cf581f1e038306243c4 P 16d7a8fd2cad4de2b23a47286e3bf735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d7a8fd2cad4de2b23a47286e3bf735, State: Initialized, Role: FOLLOWER
I20260924 00:19:44.855935 7822 raft_consensus.cc:348] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41581 } } peers { permanent_uuid: "20b755c043404eafbc6ffe99c13f08e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "d31733f022f44f95815023a3b85279af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42859 } }
I20260924 00:19:44.855983 7820 consensus_queue.cc:260] T c909ae217a6e4cf581f1e038306243c4 P 16d7a8fd2cad4de2b23a47286e3bf735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41581 } } peers { permanent_uuid: "20b755c043404eafbc6ffe99c13f08e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "d31733f022f44f95815023a3b85279af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42859 } }
I20260924 00:19:44.856027 7822 raft_consensus.cc:374] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:44.856055 7822 raft_consensus.cc:729] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b755c043404eafbc6ffe99c13f08e7, State: Initialized, Role: FOLLOWER
I20260924 00:19:44.856132 7820 ts_tablet_manager.cc:1428] T c909ae217a6e4cf581f1e038306243c4 P 16d7a8fd2cad4de2b23a47286e3bf735: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 00:19:44.856491 7821 tablet_bootstrap.cc:492] T c909ae217a6e4cf581f1e038306243c4 P d31733f022f44f95815023a3b85279af: No bootstrap required, opened a new log
I20260924 00:19:44.856115 7822 consensus_queue.cc:260] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41581 } } peers { permanent_uuid: "20b755c043404eafbc6ffe99c13f08e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "d31733f022f44f95815023a3b85279af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42859 } }
I20260924 00:19:44.856547 7821 ts_tablet_manager.cc:1397] T c909ae217a6e4cf581f1e038306243c4 P d31733f022f44f95815023a3b85279af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:19:44.856647 7822 ts_tablet_manager.cc:1428] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:44.856663 7821 raft_consensus.cc:348] T c909ae217a6e4cf581f1e038306243c4 P d31733f022f44f95815023a3b85279af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41581 } } peers { permanent_uuid: "20b755c043404eafbc6ffe99c13f08e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "d31733f022f44f95815023a3b85279af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42859 } }
I20260924 00:19:44.856719 7461 heartbeater.cc:503] Master 127.0.98.60:39793 was elected leader, sending a full tablet report...
I20260924 00:19:44.856741 7821 raft_consensus.cc:374] T c909ae217a6e4cf581f1e038306243c4 P d31733f022f44f95815023a3b85279af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:44.856766 7821 raft_consensus.cc:729] T c909ae217a6e4cf581f1e038306243c4 P d31733f022f44f95815023a3b85279af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d31733f022f44f95815023a3b85279af, State: Initialized, Role: FOLLOWER
I20260924 00:19:44.856812 7821 consensus_queue.cc:260] T c909ae217a6e4cf581f1e038306243c4 P d31733f022f44f95815023a3b85279af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41581 } } peers { permanent_uuid: "20b755c043404eafbc6ffe99c13f08e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "d31733f022f44f95815023a3b85279af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42859 } }
I20260924 00:19:44.856839 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.856371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.98.1:35921)
0924 00:19:44.856527 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:44.856667 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":115,"threads_started":1}
I20260924 00:19:44.856968 7821 ts_tablet_manager.cc:1428] T c909ae217a6e4cf581f1e038306243c4 P d31733f022f44f95815023a3b85279af: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 00:19:44.857498 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.857291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.98.2:48767)
0924 00:19:44.857419 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:44.857454 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":76,"threads_started":1}
I20260924 00:19:44.857679 7598 heartbeater.cc:503] Master 127.0.98.60:39793 was elected leader, sending a full tablet report...
I20260924 00:19:44.857767 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.857561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.98.3:53931)
0924 00:19:44.857668 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:44.857700 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":70,"threads_started":1}
I20260924 00:19:44.858319 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.857196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.98.1:50013)
0924 00:19:44.858264 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:44.858294 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1051}
I20260924 00:19:44.858517 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.858344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.98.2:47693)
0924 00:19:44.858461 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:44.858487 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260924 00:19:44.858615 7734 heartbeater.cc:503] Master 127.0.98.60:39793 was elected leader, sending a full tablet report...
I20260924 00:19:44.858914 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:44.858836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.98.3:34165)
0924 00:19:44.858866 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:44.858891 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 00:19:44.894886 7737 tablet.cc:2369] T c909ae217a6e4cf581f1e038306243c4 P d31733f022f44f95815023a3b85279af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:19:45.101533 7464 tablet.cc:2369] T c909ae217a6e4cf581f1e038306243c4 P 16d7a8fd2cad4de2b23a47286e3bf735: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:19:45.175931 7828 raft_consensus.cc:482] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:19:45.176039 7828 raft_consensus.cc:504] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41581 } } peers { permanent_uuid: "20b755c043404eafbc6ffe99c13f08e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "d31733f022f44f95815023a3b85279af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42859 } }
I20260924 00:19:45.176467 7828 leader_election.cc:288] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16d7a8fd2cad4de2b23a47286e3bf735 (127.0.98.1:41581), d31733f022f44f95815023a3b85279af (127.0.98.3:42859)
I20260924 00:19:45.179840 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:45.176647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41581 (local address 127.0.98.2:42759)
0924 00:19:45.176812 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:45.176825 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:45.176896 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:45.177112 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:45.177114 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:45.177117 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:45.177196 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:45.177198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:45.177940 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:45.177941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:45.178837 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:45.178839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:45.179560 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:45.179561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:45.179637 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:45.179642 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:19:45.179707 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 00:19:45.179752 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 00:19:45.179905 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:45.176733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42759 (local address 127.0.98.1:41581)
0924 00:19:45.176784 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:45.176787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:45.176908 (+ 121us) server_negotiation.cc:407] Connection header received
0924 00:19:45.176960 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:45.176961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:45.176975 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:45.177060 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:45.177259 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:45.177857 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:45.178939 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:45.179466 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:45.179533 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:45.179548 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:45.179739 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 00:19:45.179782 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:19:45.180049 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:45.176997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42387 (local address 127.0.98.3:42859)
0924 00:19:45.177056 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:45.177058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:45.177151 (+ 93us) server_negotiation.cc:407] Connection header received
0924 00:19:45.177201 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:45.177202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:45.177214 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:45.177337 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:45.177546 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:45.178099 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:45.179198 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:45.179649 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:45.179715 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:45.179725 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:45.179940 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 00:19:45.179983 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:19:45.180115 7415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c909ae217a6e4cf581f1e038306243c4" candidate_uuid: "20b755c043404eafbc6ffe99c13f08e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" is_pre_election: true
I20260924 00:19:45.180305 7415 raft_consensus.cc:2454] T c909ae217a6e4cf581f1e038306243c4 P 16d7a8fd2cad4de2b23a47286e3bf735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b755c043404eafbc6ffe99c13f08e7 in term 0.
I20260924 00:19:45.180292 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:45.176772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42859 (local address 127.0.98.2:42387)
0924 00:19:45.177083 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:45.177091 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:45.177143 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:45.177390 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:45.177391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:45.177392 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:45.177452 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:45.177453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:45.178209 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:45.178210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:45.179101 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:45.179103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:45.179733 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:45.179734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:45.179838 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:45.179843 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:19:45.180133 (+ 290us) client_negotiation.cc:240] Negotiation successful
0924 00:19:45.180209 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":1,"thread_start_us":129,"threads_started":1}
I20260924 00:19:45.180549 7488 leader_election.cc:302] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d7a8fd2cad4de2b23a47286e3bf735, 20b755c043404eafbc6ffe99c13f08e7; no voters:
I20260924 00:19:45.180711 7828 raft_consensus.cc:2781] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:19:45.180756 7828 raft_consensus.cc:482] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:19:45.180789 7828 raft_consensus.cc:3037] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:45.180828 7688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c909ae217a6e4cf581f1e038306243c4" candidate_uuid: "20b755c043404eafbc6ffe99c13f08e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d31733f022f44f95815023a3b85279af" is_pre_election: true
I20260924 00:19:45.180918 7688 raft_consensus.cc:2454] T c909ae217a6e4cf581f1e038306243c4 P d31733f022f44f95815023a3b85279af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b755c043404eafbc6ffe99c13f08e7 in term 0.
I20260924 00:19:45.181541 7828 raft_consensus.cc:504] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41581 } } peers { permanent_uuid: "20b755c043404eafbc6ffe99c13f08e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "d31733f022f44f95815023a3b85279af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42859 } }
I20260924 00:19:45.181685 7828 leader_election.cc:288] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7 [CANDIDATE]: Term 1 election: Requested vote from peers 16d7a8fd2cad4de2b23a47286e3bf735 (127.0.98.1:41581), d31733f022f44f95815023a3b85279af (127.0.98.3:42859)
I20260924 00:19:45.181866 7415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c909ae217a6e4cf581f1e038306243c4" candidate_uuid: "20b755c043404eafbc6ffe99c13f08e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d7a8fd2cad4de2b23a47286e3bf735"
I20260924 00:19:45.181913 7688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c909ae217a6e4cf581f1e038306243c4" candidate_uuid: "20b755c043404eafbc6ffe99c13f08e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d31733f022f44f95815023a3b85279af"
I20260924 00:19:45.181962 7415 raft_consensus.cc:3037] T c909ae217a6e4cf581f1e038306243c4 P 16d7a8fd2cad4de2b23a47286e3bf735 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:45.181974 7688 raft_consensus.cc:3037] T c909ae217a6e4cf581f1e038306243c4 P d31733f022f44f95815023a3b85279af [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:45.182617 7688 raft_consensus.cc:2454] T c909ae217a6e4cf581f1e038306243c4 P d31733f022f44f95815023a3b85279af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b755c043404eafbc6ffe99c13f08e7 in term 1.
I20260924 00:19:45.182679 7415 raft_consensus.cc:2454] T c909ae217a6e4cf581f1e038306243c4 P 16d7a8fd2cad4de2b23a47286e3bf735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b755c043404eafbc6ffe99c13f08e7 in term 1.
I20260924 00:19:45.182845 7488 leader_election.cc:302] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d7a8fd2cad4de2b23a47286e3bf735, 20b755c043404eafbc6ffe99c13f08e7; no voters:
I20260924 00:19:45.182938 7828 raft_consensus.cc:2781] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:45.183105 7828 raft_consensus.cc:686] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7 [term 1 LEADER]: Becoming Leader. State: Replica: 20b755c043404eafbc6ffe99c13f08e7, State: Running, Role: LEADER
I20260924 00:19:45.183308 7828 consensus_queue.cc:237] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41581 } } peers { permanent_uuid: "20b755c043404eafbc6ffe99c13f08e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } } peers { permanent_uuid: "d31733f022f44f95815023a3b85279af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42859 } }
I20260924 00:19:45.183501 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:45.183319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.98.2:51579)
0924 00:19:45.183377 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:45.183468 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 00:19:45.184042 7281 catalog_manager.cc:5495] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20b755c043404eafbc6ffe99c13f08e7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "20b755c043404eafbc6ffe99c13f08e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b755c043404eafbc6ffe99c13f08e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d31733f022f44f95815023a3b85279af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42859 } health_report { overall_health: UNKNOWN } } }
I20260924 00:19:45.205564 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:45.202787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57828 (local address 127.0.98.2:33353)
0924 00:19:45.202866 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:45.202868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:45.202879 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:19:45.202924 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:45.202925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:45.202949 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:45.203048 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:45.203204 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:45.203742 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:45.204608 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:45.204839 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:45.204953 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:45.205122 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:45.205297 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:45.205385 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:45.205445 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 00:19:45.205469 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 00:19:45.206740 7688 raft_consensus.cc:1261] T c909ae217a6e4cf581f1e038306243c4 P d31733f022f44f95815023a3b85279af [term 1 FOLLOWER]: Refusing update from remote peer 20b755c043404eafbc6ffe99c13f08e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:45.206961 7415 raft_consensus.cc:1261] T c909ae217a6e4cf581f1e038306243c4 P 16d7a8fd2cad4de2b23a47286e3bf735 [term 1 FOLLOWER]: Refusing update from remote peer 20b755c043404eafbc6ffe99c13f08e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:45.207006 7833 consensus_queue.cc:1035] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d31733f022f44f95815023a3b85279af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:45.207321 7833 consensus_queue.cc:1035] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d7a8fd2cad4de2b23a47286e3bf735" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:45.208024 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:45.207891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.98.3:52741)
0924 00:19:45.207950 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:45.207989 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 00:19:45.208434 7836 mvcc.cc:205] Tried to move back new op lower bound from 7332696822604791808 to 7332696822511517696. Current Snapshot: MvccSnapshot[applied={T|T < 7332696822604791808}]
I20260924 00:19:45.208899 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:45.208780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.98.3:39063)
0924 00:19:45.208837 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:45.208867 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 00:19:45.209147 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:45.208868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.98.2:54937)
0924 00:19:45.209086 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:45.209117 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
I20260924 00:19:45.209780 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:45.209662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.98.1:51811)
0924 00:19:45.209702 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:45.209738 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 00:19:45.210896 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:45.210792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.98.2:52369)
0924 00:19:45.210835 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:45.210869 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 00:19:45.211355 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:45.211233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35493 (local address 127.0.98.1:55593)
0924 00:19:45.211304 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:45.211329 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 00:19:45.212632 7281 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:45.212738 7281 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35568:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:45.214161 7281 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 152061622ae94a0ea8abcd158ad96c05: Sending DeleteTablet for 3 replicas of tablet c909ae217a6e4cf581f1e038306243c4
I20260924 00:19:45.214507 7395 tablet_service.cc:1514] Processing DeleteTablet for tablet c909ae217a6e4cf581f1e038306243c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:45 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47612
I20260924 00:19:45.214630 7668 tablet_service.cc:1514] Processing DeleteTablet for tablet c909ae217a6e4cf581f1e038306243c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:45 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:50904
I20260924 00:19:45.214720 7532 tablet_service.cc:1514] Processing DeleteTablet for tablet c909ae217a6e4cf581f1e038306243c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:45 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57820
I20260924 00:19:45.214825 7846 tablet_replica.cc:330] T c909ae217a6e4cf581f1e038306243c4 P 16d7a8fd2cad4de2b23a47286e3bf735: stopping tablet replica
I20260924 00:19:45.214891 7846 raft_consensus.cc:2229] T c909ae217a6e4cf581f1e038306243c4 P 16d7a8fd2cad4de2b23a47286e3bf735 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:45.214941 7846 raft_consensus.cc:2258] T c909ae217a6e4cf581f1e038306243c4 P 16d7a8fd2cad4de2b23a47286e3bf735 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:45.215042 7847 tablet_replica.cc:330] T c909ae217a6e4cf581f1e038306243c4 P d31733f022f44f95815023a3b85279af: stopping tablet replica
I20260924 00:19:45.215071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 7246
I20260924 00:19:45.215124 7847 raft_consensus.cc:2229] T c909ae217a6e4cf581f1e038306243c4 P d31733f022f44f95815023a3b85279af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:45.215174 7847 raft_consensus.cc:2258] T c909ae217a6e4cf581f1e038306243c4 P d31733f022f44f95815023a3b85279af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:45.215688 7848 tablet_replica.cc:330] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7: stopping tablet replica
I20260924 00:19:45.215744 7848 raft_consensus.cc:2229] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:19:45.215837 7848 raft_consensus.cc:2258] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:45.216588 7847 ts_tablet_manager.cc:1905] T c909ae217a6e4cf581f1e038306243c4 P d31733f022f44f95815023a3b85279af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:45.218313 7848 ts_tablet_manager.cc:1905] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:45.218634 7847 ts_tablet_manager.cc:1918] T c909ae217a6e4cf581f1e038306243c4 P d31733f022f44f95815023a3b85279af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:19:45.218755 7847 log.cc:1196] T c909ae217a6e4cf581f1e038306243c4 P d31733f022f44f95815023a3b85279af: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/c909ae217a6e4cf581f1e038306243c4
I20260924 00:19:45.219126 7847 ts_tablet_manager.cc:1939] T c909ae217a6e4cf581f1e038306243c4 P d31733f022f44f95815023a3b85279af: Deleting consensus metadata
I20260924 00:19:45.219542 7269 catalog_manager.cc:4841] TS d31733f022f44f95815023a3b85279af (127.0.98.3:42859): tablet c909ae217a6e4cf581f1e038306243c4 (table test-table [id=47c168c4ab4d4a62829e497a400dda10]) successfully deleted
I20260924 00:19:45.220134 7848 ts_tablet_manager.cc:1918] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:19:45.220211 7848 log.cc:1196] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790209154318147-392-0/minicluster-data/ts-1/wal/wals/c909ae217a6e4cf581f1e038306243c4
I20260924 00:19:45.220432 7848 ts_tablet_manager.cc:1939] T c909ae217a6e4cf581f1e038306243c4 P 20b755c043404eafbc6ffe99c13f08e7: Deleting consensus metadata
I20260924 00:19:45.221755 7269 catalog_manager.cc:4841] TS 20b755c043404eafbc6ffe99c13f08e7 (127.0.98.2:33353): tablet c909ae217a6e4cf581f1e038306243c4 (table test-table [id=47c168c4ab4d4a62829e497a400dda10]) successfully deleted
W20260924 00:19:45.224642 7800 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:45.224496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41581 (local address 127.0.0.1:47626)
0924 00:19:45.224558 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:45.224588 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41581: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 00:19:45.224784 7269 catalog_manager.cc:4584] TS 16d7a8fd2cad4de2b23a47286e3bf735 (127.0.98.1:41581): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c909ae217a6e4cf581f1e038306243c4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41581: connect: Connection reset by peer (error 104)
I20260924 00:19:45.224885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 7469
I20260924 00:19:45.230037 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 7605
I20260924 00:19:45.234941 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 6960
I20260924 00:19:45.240087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 7148
2026-09-24T00:19:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5210 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:19:45 dist-test-slave-777x krb5kdc[7851](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:19:45 dist-test-slave-777x krb5kdc[7851](info): set up 2 sockets
Sep 24 00:19:45 dist-test-slave-777x krb5kdc[7851](info): commencing operation
krb5kdc: starting...
W20260924 00:19:46.381099 392 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:19:46 dist-test-slave-777x krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:19:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:19:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 00:19:46.507117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43107
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:46.596064 7864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:46.596326 7864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:46.596354 7864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:46.597656 7864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:19:46.597707 7864 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 00:19:46.597723 7864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:46.597738 7864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:46.597752 7864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:19:46.597766 7864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:19:46.599364 7864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43107
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:46.599586 7864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:46.599773 7864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:46.604948 7870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:46.604965 7872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:46.604952 7869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:46.605191 7864 server_base.cc:1034] running on GCE node
I20260924 00:19:46.605523 7864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:46.605844 7864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:46.607043 7864 hybrid_clock.cc:648] HybridClock initialized: now 1790209186607017 us; error 34 us; skew 500 ppm
Sep 24 00:19:46 dist-test-slave-777x krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209186, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:46.609826 7864 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 00:19:46.610298 7864 webserver.cc:460] Webserver started at http://127.0.98.62:37967/ using document root <none> and password file <none>
I20260924 00:19:46.610548 7864 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:46.610620 7864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:46.610793 7864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:46.612066 7864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "3244e3d5b0c84d68ae59b5f905e995be"
format_stamp: "Formatted at 2026-09-24 00:19:46 on dist-test-slave-777x"
server_key: "47600610cc8565e545983d807ba29024"
server_key_iv: "b805990d87390ebadb50c445ab18f108"
server_key_version: "encryptionkey@0"
I20260924 00:19:46.612568 7864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "3244e3d5b0c84d68ae59b5f905e995be"
format_stamp: "Formatted at 2026-09-24 00:19:46 on dist-test-slave-777x"
server_key: "47600610cc8565e545983d807ba29024"
server_key_iv: "b805990d87390ebadb50c445ab18f108"
server_key_version: "encryptionkey@0"
I20260924 00:19:46.614094 7864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:19:46.614889 7879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:46.615127 7864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:46.615242 7864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "3244e3d5b0c84d68ae59b5f905e995be"
format_stamp: "Formatted at 2026-09-24 00:19:46 on dist-test-slave-777x"
server_key: "47600610cc8565e545983d807ba29024"
server_key_iv: "b805990d87390ebadb50c445ab18f108"
server_key_version: "encryptionkey@0"
I20260924 00:19:46.615396 7864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:46.631526 7864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:46.634140 7864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:46.634306 7864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:46.639107 7864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43107
I20260924 00:19:46.639150 7941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43107 every 8 connection(s)
I20260924 00:19:46.639484 7864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:19:46.640106 7942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:46.642091 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 7864
I20260924 00:19:46.642168 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:19:46.642382 392 external_mini_cluster.cc:1428] Setting key 6d4a2c3ae6af4fcf6fb217aa5188ba0e
I20260924 00:19:46.642822 7942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be: Bootstrap starting.
I20260924 00:19:46.643580 7942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:46.643910 7942 log.cc:824] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:46.644811 7942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be: No bootstrap required, opened a new log
I20260924 00:19:46.647347 7942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3244e3d5b0c84d68ae59b5f905e995be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43107 } }
I20260924 00:19:46.647518 7942 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:46.647544 7942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3244e3d5b0c84d68ae59b5f905e995be, State: Initialized, Role: FOLLOWER
I20260924 00:19:46.647692 7942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3244e3d5b0c84d68ae59b5f905e995be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43107 } }
I20260924 00:19:46.647774 7942 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:19:46.647799 7942 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:19:46.647879 7942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:46.648695 7942 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3244e3d5b0c84d68ae59b5f905e995be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43107 } }
I20260924 00:19:46.648844 7942 leader_election.cc:302] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3244e3d5b0c84d68ae59b5f905e995be; no voters:
I20260924 00:19:46.649052 7942 leader_election.cc:288] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:19:46.649116 7947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:46.649281 7947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be [term 1 LEADER]: Becoming Leader. State: Replica: 3244e3d5b0c84d68ae59b5f905e995be, State: Running, Role: LEADER
I20260924 00:19:46.649359 7942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:19:46.649384 7947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3244e3d5b0c84d68ae59b5f905e995be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43107 } }
I20260924 00:19:46.649739 7947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3244e3d5b0c84d68ae59b5f905e995be. Latest consensus state: current_term: 1 leader_uuid: "3244e3d5b0c84d68ae59b5f905e995be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3244e3d5b0c84d68ae59b5f905e995be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43107 } } }
I20260924 00:19:46.649830 7947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:46.649747 7948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3244e3d5b0c84d68ae59b5f905e995be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3244e3d5b0c84d68ae59b5f905e995be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43107 } } }
I20260924 00:19:46.650034 7953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:19:46.650051 7948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be [sys.catalog]: This master's current role is: LEADER
Sep 24 00:19:46 dist-test-slave-777x krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:19:46.650557 7953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:19:46.654479 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:46.643001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32782 (local address 127.0.98.62:43107)
0924 00:19:46.643200 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:46.643211 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:46.643251 (+ 40us) server_negotiation.cc:407] Connection header received
0924 00:19:46.643906 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:46.643914 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:46.643992 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:46.644182 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:46.645123 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:46.645997 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:46.647179 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:46.647747 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:46.647901 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:46.650966 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:46.650970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:46.650970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:46.650983 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:46.652899 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:46.653340 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:46.653344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:46.653344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:46.653436 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:46.653650 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:46.653651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:46.653651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:46.653890 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:46.654073 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:46.654162 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 00:19:46.654300 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260924 00:19:46.656564 7953 catalog_manager.cc:1347] Generated new cluster ID: 9e4c294bf1a749828046ddd39e75ad81
I20260924 00:19:46.656637 7953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:19:46.675031 7953 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:19:46.675627 7953 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:19:46.682080 7953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be: Generated new TSK 0
I20260924 00:19:46.682340 7953 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 00:19:46.716286 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43107
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:46.798094 7968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:46.798314 7968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:46.798342 7968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:46.799469 7968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:46.799507 7968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:46.799527 7968 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:19:46.799546 7968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:19:46.801100 7968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:46.801317 7968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:46.801502 7968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:19:46.806656 7968 server_base.cc:1034] running on GCE node
W20260924 00:19:46.806612 7976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:46.806612 7973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:46.806645 7974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:46.806996 7968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:46.807237 7968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:46.808386 7968 hybrid_clock.cc:648] HybridClock initialized: now 1790209186808339 us; error 68 us; skew 500 ppm
Sep 24 00:19:46 dist-test-slave-777x krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209186, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:46.810900 7968 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 00:19:46.811486 7968 webserver.cc:460] Webserver started at http://127.0.98.1:42753/ using document root <none> and password file <none>
I20260924 00:19:46.811728 7968 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:46.811793 7968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:46.811929 7968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:46.813067 7968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "2331dfca9ae24769b1aba1eef5e3114a"
format_stamp: "Formatted at 2026-09-24 00:19:46 on dist-test-slave-777x"
server_key: "1fc7ad0e15226365097a2540b0fca810"
server_key_iv: "048028ea7e2b644bb837ffc3d7c463c6"
server_key_version: "encryptionkey@0"
I20260924 00:19:46.813428 7968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2331dfca9ae24769b1aba1eef5e3114a"
format_stamp: "Formatted at 2026-09-24 00:19:46 on dist-test-slave-777x"
server_key: "1fc7ad0e15226365097a2540b0fca810"
server_key_iv: "048028ea7e2b644bb837ffc3d7c463c6"
server_key_version: "encryptionkey@0"
I20260924 00:19:46.814778 7968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 00:19:46.815596 7983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:46.815789 7968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:46.815887 7968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "2331dfca9ae24769b1aba1eef5e3114a"
format_stamp: "Formatted at 2026-09-24 00:19:46 on dist-test-slave-777x"
server_key: "1fc7ad0e15226365097a2540b0fca810"
server_key_iv: "048028ea7e2b644bb837ffc3d7c463c6"
server_key_version: "encryptionkey@0"
I20260924 00:19:46.815970 7968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:46.828636 7968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:46.830643 7968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:46.830806 7968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:46.831398 7968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:46.831465 7968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:46.831499 7968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:46.831523 7968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 00:19:46 dist-test-slave-777x krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209186, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:19:46.838986 7968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35223
I20260924 00:19:46.839399 7968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:19:46.839406 8098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35223 every 8 connection(s)
I20260924 00:19:46.841344 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 7968
I20260924 00:19:46.841467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:19:46.841688 392 external_mini_cluster.cc:1428] Setting key 35ed87243f08494f23500f6a9ad6823a
I20260924 00:19:46.841750 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:46.831835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33131 (local address 127.0.98.62:43107)
0924 00:19:46.831915 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:46.831923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:46.832186 (+ 263us) server_negotiation.cc:407] Connection header received
0924 00:19:46.832754 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:46.832756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:46.832771 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:46.832868 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:46.833636 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:46.834194 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:46.835325 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:46.835627 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:46.835708 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:46.838479 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:46.838481 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:46.838481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:46.838490 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:46.840309 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:46.840726 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:46.840727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:46.840727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:46.840779 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:46.841021 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:46.841022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:46.841022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:46.841124 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:46.841175 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:46.841597 (+ 422us) server_negotiation.cc:299] Negotiation successful
0924 00:19:46.841667 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 00:19:46.842025 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:46.831832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43107 (local address 127.0.98.1:33131)
0924 00:19:46.832027 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:46.832070 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:46.832651 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:46.832924 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:46.832928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:46.833227 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:46.833509 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:46.833515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:46.834294 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:46.834295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:46.835212 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:46.835214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:46.835729 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:46.835730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:46.835902 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:46.836485 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:46.836499 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:46.838372 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:46.840435 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:46.840441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:46.840444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:46.840655 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:46.840843 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:46.840844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:46.840844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:46.840965 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:46.841211 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:46.841219 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:46.841425 (+ 206us) client_negotiation.cc:769] Sending connection context
0924 00:19:46.841572 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 00:19:46.841760 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":124,"threads_started":1}
I20260924 00:19:46.842690 8099 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43107
I20260924 00:19:46.842789 8099 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:46.843006 8099 heartbeater.cc:511] Master 127.0.98.62:43107 requested a full tablet report, sending...
I20260924 00:19:46.843834 7896 ts_manager.cc:194] Registered new tserver with Master: 2331dfca9ae24769b1aba1eef5e3114a (127.0.98.1:35223)
I20260924 00:19:46.844544 7896 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33131
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 00:19:46.851650 8106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:46.843696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43107 (local address 127.0.98.1:47105)
0924 00:19:46.844101 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:46.844113 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:46.844190 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:46.844870 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:46.844871 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:46.845078 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:46.845143 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:46.845145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:46.845922 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:46.845923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:46.846840 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:46.846842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:46.847194 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:46.847195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:46.847266 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:46.847814 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:46.847820 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:46.848279 (+ 459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:46.850191 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:46.850192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:46.850192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:46.850541 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:46.850897 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:46.850899 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:46.850899 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:46.850988 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:46.851293 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:46.851294 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:46.851339 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 00:19:46.851423 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 00:19:46.851513 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":68,"threads_started":1}
I20260924 00:19:46.851711 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:46.844743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47105 (local address 127.0.98.62:43107)
0924 00:19:46.844778 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:46.844779 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:46.844787 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:19:46.844805 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:46.844806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:46.844814 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:46.844864 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:46.845223 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:46.845804 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:46.846921 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:46.847113 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:46.847174 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:46.848391 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:46.848393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:46.848393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:46.848400 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:46.850098 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:46.850733 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:46.850734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:46.850734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:46.850794 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:46.851065 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:46.851066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:46.851066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:46.851200 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:46.851274 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:46.851482 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 00:19:46.851616 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 00:19:46.870810 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43107
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:46.952101 8108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:46.952358 8108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:46.952400 8108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:46.953608 8108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:46.953675 8108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:46.953709 8108 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:19:46.953749 8108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:19:46.955215 8108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:46.955489 8108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:46.955719 8108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:46.961076 8114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:46.961087 8116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:46.961164 8108 server_base.cc:1034] running on GCE node
W20260924 00:19:46.961087 8113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:46.961498 8108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:46.961740 8108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:46.962891 8108 hybrid_clock.cc:648] HybridClock initialized: now 1790209186962877 us; error 25 us; skew 500 ppm
Sep 24 00:19:46 dist-test-slave-777x krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209186, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:46.965556 8108 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 00:19:46.966030 8108 webserver.cc:460] Webserver started at http://127.0.98.2:42215/ using document root <none> and password file <none>
I20260924 00:19:46.966235 8108 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:46.966276 8108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:46.966363 8108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:46.967422 8108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "99153313d5e843eb9d988459ee092574"
format_stamp: "Formatted at 2026-09-24 00:19:46 on dist-test-slave-777x"
server_key: "28f311ed1068df0791b429f866c24ba9"
server_key_iv: "54cf5fdbc67cacf424d11084a48bb495"
server_key_version: "encryptionkey@0"
I20260924 00:19:46.967734 8108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "99153313d5e843eb9d988459ee092574"
format_stamp: "Formatted at 2026-09-24 00:19:46 on dist-test-slave-777x"
server_key: "28f311ed1068df0791b429f866c24ba9"
server_key_iv: "54cf5fdbc67cacf424d11084a48bb495"
server_key_version: "encryptionkey@0"
I20260924 00:19:46.969147 8108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:19:46.969852 8123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:46.970052 8108 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:19:46.970125 8108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "99153313d5e843eb9d988459ee092574"
format_stamp: "Formatted at 2026-09-24 00:19:46 on dist-test-slave-777x"
server_key: "28f311ed1068df0791b429f866c24ba9"
server_key_iv: "54cf5fdbc67cacf424d11084a48bb495"
server_key_version: "encryptionkey@0"
I20260924 00:19:46.970181 8108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:46.998003 8108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:47.000465 8108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:47.000588 8108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:47.001051 8108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:47.001113 8108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:47.001168 8108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:47.001201 8108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 00:19:47 dist-test-slave-777x krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209186, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:19:47.008765 8108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41125
I20260924 00:19:47.008862 8238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41125 every 8 connection(s)
I20260924 00:19:47.009195 8108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:19:47.011109 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.001652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50229 (local address 127.0.98.62:43107)
0924 00:19:47.001727 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:47.001731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:47.001889 (+ 158us) server_negotiation.cc:407] Connection header received
0924 00:19:47.002385 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:47.002386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:47.002401 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:47.002484 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:47.003196 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.004030 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.005174 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.005373 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.005483 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.008357 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:47.008358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:47.008359 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:47.008367 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:47.009960 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:47.010350 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:47.010351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:47.010351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:47.010402 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:47.010577 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:47.010578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:47.010578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:47.010677 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:47.010728 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:47.010975 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 00:19:47.011043 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 00:19:47.011277 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.001609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43107 (local address 127.0.98.2:50229)
0924 00:19:47.001756 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:47.001796 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:47.002285 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:47.002599 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:47.002603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:47.002911 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:47.003118 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.003123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.004131 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.004132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.005080 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.005083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.005500 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.005501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.005683 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.006290 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:47.006304 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:47.008257 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:47.010068 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:47.010071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:47.010073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:47.010282 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:47.010443 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:47.010443 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:47.010443 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:47.010527 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:47.010737 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:47.010742 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:47.010851 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 00:19:47.010935 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 00:19:47.011084 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260924 00:19:47.011888 8239 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43107
I20260924 00:19:47.012001 8239 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:47.012220 8239 heartbeater.cc:511] Master 127.0.98.62:43107 requested a full tablet report, sending...
I20260924 00:19:47.012625 7895 ts_manager.cc:194] Registered new tserver with Master: 99153313d5e843eb9d988459ee092574 (127.0.98.2:41125)
I20260924 00:19:47.013131 7895 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50229
I20260924 00:19:47.016551 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 8108
I20260924 00:19:47.016654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:19:47.016834 392 external_mini_cluster.cc:1428] Setting key 02d93bc73a42f52dbb9e03d24ce86183
I20260924 00:19:47.018482 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.012992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52113 (local address 127.0.98.62:43107)
0924 00:19:47.013028 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:47.013030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:47.013157 (+ 127us) server_negotiation.cc:407] Connection header received
0924 00:19:47.013216 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:47.013217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:47.013228 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:47.013283 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:47.013639 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.014086 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.014750 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.014918 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.014968 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.015828 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:47.015828 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:47.015829 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:47.015833 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:47.017271 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:47.017633 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:47.017634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:47.017634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:47.017682 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:47.017885 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:47.017885 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:47.017886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:47.018030 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:47.018134 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:47.018304 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 00:19:47.018373 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 00:19:47.018553 8245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.012920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43107 (local address 127.0.98.2:52113)
0924 00:19:47.013082 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:47.013093 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:47.013156 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:47.013288 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:47.013289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:47.013451 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:47.013560 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.013562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.014175 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.014176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.014680 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.014681 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.014982 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.014982 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.015044 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.015484 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:47.015489 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:47.015761 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:47.017341 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:47.017342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:47.017342 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:47.017562 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:47.017747 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:47.017748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:47.017748 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:47.017819 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:47.018130 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:47.018131 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:47.018182 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 00:19:47.018269 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 00:19:47.018382 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 00:19:47.043035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43107
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:47.124032 8248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:47.124342 8248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:47.124384 8248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:47.125562 8248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:47.125604 8248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:47.125625 8248 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 00:19:47.125643 8248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:19:47.127059 8248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:47.127236 8248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:47.127405 8248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:47.132969 8256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:47.132989 8253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:47.132988 8254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:47.133237 8248 server_base.cc:1034] running on GCE node
I20260924 00:19:47.133441 8248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:47.133651 8248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:47.134832 8248 hybrid_clock.cc:648] HybridClock initialized: now 1790209187134810 us; error 32 us; skew 500 ppm
Sep 24 00:19:47 dist-test-slave-777x krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209187, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:47.138003 8248 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 00:19:47.138522 8248 webserver.cc:460] Webserver started at http://127.0.98.3:34621/ using document root <none> and password file <none>
I20260924 00:19:47.138782 8248 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:47.138857 8248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:47.139026 8248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:47.140300 8248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "fac83bdbf50345c2b8df30f54cddd7ab"
format_stamp: "Formatted at 2026-09-24 00:19:47 on dist-test-slave-777x"
server_key: "b83b8a3b3a8a3a1bfb0b4946dcc8ec75"
server_key_iv: "a5180b502b8f35c7e25a4bbcb3734dc4"
server_key_version: "encryptionkey@0"
I20260924 00:19:47.140708 8248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fac83bdbf50345c2b8df30f54cddd7ab"
format_stamp: "Formatted at 2026-09-24 00:19:47 on dist-test-slave-777x"
server_key: "b83b8a3b3a8a3a1bfb0b4946dcc8ec75"
server_key_iv: "a5180b502b8f35c7e25a4bbcb3734dc4"
server_key_version: "encryptionkey@0"
I20260924 00:19:47.142189 8248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 00:19:47.143047 8263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:47.143234 8248 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:19:47.143354 8248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "fac83bdbf50345c2b8df30f54cddd7ab"
format_stamp: "Formatted at 2026-09-24 00:19:47 on dist-test-slave-777x"
server_key: "b83b8a3b3a8a3a1bfb0b4946dcc8ec75"
server_key_iv: "a5180b502b8f35c7e25a4bbcb3734dc4"
server_key_version: "encryptionkey@0"
I20260924 00:19:47.143496 8248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:47.155953 8248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:47.158727 8248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:47.158905 8248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:47.159461 8248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:47.159531 8248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:47.159581 8248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:47.159607 8248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:47.166436 8248 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36929
I20260924 00:19:47.166831 8248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:19:47.166502 8378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36929 every 8 connection(s)
Sep 24 00:19:47 dist-test-slave-777x krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209187, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:19:47.168391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 8248
I20260924 00:19:47.168498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:19:47.168738 392 external_mini_cluster.cc:1428] Setting key 9211a01110a01031d121636cf6e2c65f
I20260924 00:19:47.171497 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.160014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48151 (local address 127.0.98.62:43107)
0924 00:19:47.160060 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:47.160062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:47.160368 (+ 306us) server_negotiation.cc:407] Connection header received
0924 00:19:47.161058 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:47.161059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:47.161074 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:47.161143 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:47.162142 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.163137 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.165035 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.165386 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.165479 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.168397 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:47.168398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:47.168398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:47.168405 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:47.170139 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:47.170663 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:47.170663 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:47.170664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:47.170714 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:47.170947 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:47.170948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:47.170948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:47.171065 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:47.171112 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:47.171372 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 00:19:47.171433 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:19:47.171836 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.160003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43107 (local address 127.0.98.3:48151)
0924 00:19:47.160155 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:47.160277 (+ 122us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:47.160843 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:47.161268 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:47.161273 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:47.161722 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:47.162037 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.162044 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.163258 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.163259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.164898 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.164902 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.165501 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.165502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.165704 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.166476 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:47.166488 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:47.168269 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:47.170219 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:47.170225 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:47.170227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:47.170585 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:47.170770 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:47.170770 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:47.170771 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:47.170884 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:47.171121 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:47.171125 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:47.171251 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 00:19:47.171335 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 00:19:47.171581 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260924 00:19:47.172361 8379 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43107
I20260924 00:19:47.172457 8379 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:47.172665 8379 heartbeater.cc:511] Master 127.0.98.62:43107 requested a full tablet report, sending...
I20260924 00:19:47.173071 7895 ts_manager.cc:194] Registered new tserver with Master: fac83bdbf50345c2b8df30f54cddd7ab (127.0.98.3:36929)
I20260924 00:19:47.173444 7895 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48151
I20260924 00:19:47.180073 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.173654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33527 (local address 127.0.98.62:43107)
0924 00:19:47.173699 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:47.173701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:47.173760 (+ 59us) server_negotiation.cc:407] Connection header received
0924 00:19:47.173840 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:47.173841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:47.173852 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:47.173913 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:47.174333 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.174863 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.175646 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.175882 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.175975 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.177086 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:47.177087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:47.177087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:47.177095 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:47.178729 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:47.179154 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:47.179155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:47.179156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:47.179222 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:47.179469 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:47.179470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:47.179470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:47.179604 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:47.179691 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:47.179858 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 00:19:47.179963 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:19:47.180091 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.173550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43107 (local address 127.0.98.3:33527)
0924 00:19:47.173704 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:47.173717 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:47.173778 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:47.173930 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:47.173931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:47.174180 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:47.174249 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.174251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.174964 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.174965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.175535 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.175537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.175979 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.175980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.176054 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.176602 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:47.176607 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:47.176945 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:47.178855 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:47.178856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:47.178857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:47.179051 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:47.179303 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:47.179304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:47.179305 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:47.179378 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:47.179680 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:47.179681 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:47.179732 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 00:19:47.179831 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 00:19:47.179977 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260924 00:19:47.180876 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 00:19:47 dist-test-slave-777x krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209187, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 00:19:47 dist-test-slave-777x krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:47 dist-test-slave-777x krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 00:19:47.203202 7945 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:47.197387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32796 (local address 127.0.98.62:43107)
0924 00:19:47.197461 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:47.197463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:47.197470 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:19:47.197496 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:47.197496 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:47.197516 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:47.197565 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:47.198173 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.198758 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.199471 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.199681 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.199744 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.203078 (+ 3334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32796: BlockingRecv error: recv got EOF from 127.0.0.1:32796 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 00:19:47 dist-test-slave-777x krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209187, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:19:47.210860 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.203983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32802 (local address 127.0.98.62:43107)
0924 00:19:47.204050 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:47.204059 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:47.204068 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:19:47.204115 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:47.204115 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:47.204124 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:47.204225 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:47.204649 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.205161 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.205837 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.206049 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.206117 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.208668 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:47.208669 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:47.208669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:47.208676 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:47.209874 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:47.210187 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:47.210187 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:47.210188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:47.210237 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:47.210407 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:47.210408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:47.210408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:47.210514 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:47.210587 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:47.210737 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 00:19:47.210794 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 00:19:47.212749 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:19:47.214485 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:19:47.221503 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.216830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36929 (local address 127.0.0.1:36780)
0924 00:19:47.216968 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:47.216984 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:47.217058 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:47.217814 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:47.217814 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:47.217816 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:47.217855 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.217857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.219279 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.219280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.220366 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.220369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.221174 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.221175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.221255 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.221270 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 00:19:47.221352 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 00:19:47.221412 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260924 00:19:47.221529 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.217370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36780 (local address 127.0.98.3:36929)
0924 00:19:47.217642 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:47.217648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:47.217661 (+ 13us) server_negotiation.cc:407] Connection header received
0924 00:19:47.217704 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:47.217706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:47.217722 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:47.217815 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:47.217973 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.219153 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.220451 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.221083 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.221157 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.221228 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:47.221375 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 00:19:47.221412 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":132,"threads_started":1}
I20260924 00:19:47.222028 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.217057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41125 (local address 127.0.0.1:59752)
0924 00:19:47.217177 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:47.217186 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:47.217228 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:47.218163 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:47.218164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:47.218166 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:47.218208 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.218210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.219476 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.219477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.220816 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.220818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.221786 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.221787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.221848 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.221852 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:19:47.221896 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 00:19:47.221962 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":86,"threads_started":1}
I20260924 00:19:47.222378 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.216790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35223 (local address 127.0.0.1:42322)
0924 00:19:47.216932 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:47.216980 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:47.217064 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:47.217571 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:47.217575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:47.217577 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:47.217642 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.217644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.219008 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.219009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.220711 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.220714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.222227 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.222228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.222271 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.222274 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:19:47.222311 (+ 37us) client_negotiation.cc:240] Negotiation successful
0924 00:19:47.222338 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":26,"thread_start_us":92,"threads_started":1}
I20260924 00:19:47.222335 8312 tablet_service.cc:1467] Processing CreateTablet for tablet f7aa4ff744a745b58a8dea6ee798f782 (DEFAULT_TABLE table=test-table [id=bdb44f00732e418d8f23875e2f46a7ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:47.222651 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.217221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42322 (local address 127.0.98.1:35223)
0924 00:19:47.217381 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:47.217385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:47.217393 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:19:47.217425 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:47.217444 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:47.217461 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:47.217563 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:47.217718 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.218900 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.220802 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.221539 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.221991 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.222481 (+ 490us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:47.222518 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 00:19:47.222557 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20260924 00:19:47.222703 8312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7aa4ff744a745b58a8dea6ee798f782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:47.223219 8033 tablet_service.cc:1467] Processing CreateTablet for tablet f7aa4ff744a745b58a8dea6ee798f782 (DEFAULT_TABLE table=test-table [id=bdb44f00732e418d8f23875e2f46a7ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:47.223512 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7aa4ff744a745b58a8dea6ee798f782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:47.224519 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.217611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59752 (local address 127.0.98.2:41125)
0924 00:19:47.218000 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:47.218005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:47.218017 (+ 12us) server_negotiation.cc:407] Connection header received
0924 00:19:47.218056 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:47.218058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:47.218073 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:47.218159 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:47.218295 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.219394 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.220886 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.221699 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.221776 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.224351 (+ 2575us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:47.224381 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 00:19:47.224424 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":82,"threads_started":1}
I20260924 00:19:47.225147 8173 tablet_service.cc:1467] Processing CreateTablet for tablet f7aa4ff744a745b58a8dea6ee798f782 (DEFAULT_TABLE table=test-table [id=bdb44f00732e418d8f23875e2f46a7ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:47.225500 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7aa4ff744a745b58a8dea6ee798f782. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:47.226629 8407 tablet_bootstrap.cc:492] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab: Bootstrap starting.
I20260924 00:19:47.226908 8408 tablet_bootstrap.cc:492] T f7aa4ff744a745b58a8dea6ee798f782 P 2331dfca9ae24769b1aba1eef5e3114a: Bootstrap starting.
I20260924 00:19:47.227309 8407 tablet_bootstrap.cc:654] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:47.227589 8408 tablet_bootstrap.cc:654] T f7aa4ff744a745b58a8dea6ee798f782 P 2331dfca9ae24769b1aba1eef5e3114a: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:47.227607 8407 log.cc:824] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:47.227957 8408 log.cc:824] T f7aa4ff744a745b58a8dea6ee798f782 P 2331dfca9ae24769b1aba1eef5e3114a: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:47.228366 8407 tablet_bootstrap.cc:492] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab: No bootstrap required, opened a new log
I20260924 00:19:47.228441 8407 ts_tablet_manager.cc:1397] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:19:47.228901 8408 tablet_bootstrap.cc:492] T f7aa4ff744a745b58a8dea6ee798f782 P 2331dfca9ae24769b1aba1eef5e3114a: No bootstrap required, opened a new log
I20260924 00:19:47.228978 8408 ts_tablet_manager.cc:1397] T f7aa4ff744a745b58a8dea6ee798f782 P 2331dfca9ae24769b1aba1eef5e3114a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 00:19:47.229758 8411 tablet_bootstrap.cc:492] T f7aa4ff744a745b58a8dea6ee798f782 P 99153313d5e843eb9d988459ee092574: Bootstrap starting.
I20260924 00:19:47.230084 8407 raft_consensus.cc:348] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.230257 8407 raft_consensus.cc:374] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:47.230300 8407 raft_consensus.cc:729] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac83bdbf50345c2b8df30f54cddd7ab, State: Initialized, Role: FOLLOWER
I20260924 00:19:47.230429 8407 consensus_queue.cc:260] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.230537 8411 tablet_bootstrap.cc:654] T f7aa4ff744a745b58a8dea6ee798f782 P 99153313d5e843eb9d988459ee092574: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:47.230753 8407 ts_tablet_manager.cc:1428] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:19:47.230887 8379 heartbeater.cc:503] Master 127.0.98.62:43107 was elected leader, sending a full tablet report...
I20260924 00:19:47.231168 8411 log.cc:824] T f7aa4ff744a745b58a8dea6ee798f782 P 99153313d5e843eb9d988459ee092574: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:47.231227 8408 raft_consensus.cc:348] T f7aa4ff744a745b58a8dea6ee798f782 P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.231405 8408 raft_consensus.cc:374] T f7aa4ff744a745b58a8dea6ee798f782 P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:47.231449 8408 raft_consensus.cc:729] T f7aa4ff744a745b58a8dea6ee798f782 P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2331dfca9ae24769b1aba1eef5e3114a, State: Initialized, Role: FOLLOWER
I20260924 00:19:47.231566 8408 consensus_queue.cc:260] T f7aa4ff744a745b58a8dea6ee798f782 P 2331dfca9ae24769b1aba1eef5e3114a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.231877 8408 ts_tablet_manager.cc:1428] T f7aa4ff744a745b58a8dea6ee798f782 P 2331dfca9ae24769b1aba1eef5e3114a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 00:19:47.232012 8411 tablet_bootstrap.cc:492] T f7aa4ff744a745b58a8dea6ee798f782 P 99153313d5e843eb9d988459ee092574: No bootstrap required, opened a new log
I20260924 00:19:47.232108 8411 ts_tablet_manager.cc:1397] T f7aa4ff744a745b58a8dea6ee798f782 P 99153313d5e843eb9d988459ee092574: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:19:47.233448 8411 raft_consensus.cc:348] T f7aa4ff744a745b58a8dea6ee798f782 P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.233628 8411 raft_consensus.cc:374] T f7aa4ff744a745b58a8dea6ee798f782 P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:47.233664 8411 raft_consensus.cc:729] T f7aa4ff744a745b58a8dea6ee798f782 P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99153313d5e843eb9d988459ee092574, State: Initialized, Role: FOLLOWER
I20260924 00:19:47.233763 8411 consensus_queue.cc:260] T f7aa4ff744a745b58a8dea6ee798f782 P 99153313d5e843eb9d988459ee092574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.234020 8411 ts_tablet_manager.cc:1428] T f7aa4ff744a745b58a8dea6ee798f782 P 99153313d5e843eb9d988459ee092574: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 00:19:47.234087 8239 heartbeater.cc:503] Master 127.0.98.62:43107 was elected leader, sending a full tablet report...
I20260924 00:19:47.234153 8099 heartbeater.cc:503] Master 127.0.98.62:43107 was elected leader, sending a full tablet report...
W20260924 00:19:47.259825 8240 tablet.cc:2369] T f7aa4ff744a745b58a8dea6ee798f782 P 99153313d5e843eb9d988459ee092574: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:19:47.340849 8100 tablet.cc:2369] T f7aa4ff744a745b58a8dea6ee798f782 P 2331dfca9ae24769b1aba1eef5e3114a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:19:47.365154 8412 raft_consensus.cc:482] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:19:47.365281 8412 raft_consensus.cc:504] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.365572 8412 leader_election.cc:288] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2331dfca9ae24769b1aba1eef5e3114a (127.0.98.1:35223), 99153313d5e843eb9d988459ee092574 (127.0.98.2:41125)
I20260924 00:19:47.369302 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.366017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33847 (local address 127.0.98.2:41125)
0924 00:19:47.366086 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:47.366089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:47.366211 (+ 122us) server_negotiation.cc:407] Connection header received
0924 00:19:47.366277 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:47.366278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:47.366289 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:47.366370 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:47.366598 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.367070 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.368333 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.368907 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.368988 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.368998 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:47.369187 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 00:19:47.369229 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 00:19:47.369302 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.365860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49835 (local address 127.0.98.1:35223)
0924 00:19:47.365917 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:47.365920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:47.365936 (+ 16us) server_negotiation.cc:407] Connection header received
0924 00:19:47.365992 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:47.365994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:47.366006 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:47.366079 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:47.366254 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.366809 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.368257 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.368902 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.368988 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.368998 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:47.369187 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 00:19:47.369229 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 00:19:47.369318 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.365793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35223 (local address 127.0.98.3:49835)
0924 00:19:47.365853 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:47.365874 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:47.365923 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:47.366100 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:47.366101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:47.366105 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:47.366173 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.366175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.366905 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.366906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.368102 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.368104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.369005 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.369006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.369100 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.369106 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 00:19:47.369168 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 00:19:47.369209 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":5}
I20260924 00:19:47.369335 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.365942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41125 (local address 127.0.98.3:33847)
0924 00:19:47.366146 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:47.366153 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:47.366224 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:47.366441 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:47.366442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:47.366443 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:47.366539 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.366541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.367169 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.367170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.368264 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.368266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.369005 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.369006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.369100 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.369106 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 00:19:47.369168 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 00:19:47.369209 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":18,"thread_start_us":124,"threads_started":1}
I20260924 00:19:47.369701 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7aa4ff744a745b58a8dea6ee798f782" candidate_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99153313d5e843eb9d988459ee092574" is_pre_election: true
I20260924 00:19:47.369889 8193 raft_consensus.cc:2454] T f7aa4ff744a745b58a8dea6ee798f782 P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac83bdbf50345c2b8df30f54cddd7ab in term 0.
I20260924 00:19:47.370158 8265 leader_election.cc:302] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99153313d5e843eb9d988459ee092574, fac83bdbf50345c2b8df30f54cddd7ab; no voters:
I20260924 00:19:47.370136 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7aa4ff744a745b58a8dea6ee798f782" candidate_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2331dfca9ae24769b1aba1eef5e3114a" is_pre_election: true
I20260924 00:19:47.370276 8053 raft_consensus.cc:2454] T f7aa4ff744a745b58a8dea6ee798f782 P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac83bdbf50345c2b8df30f54cddd7ab in term 0.
I20260924 00:19:47.370323 8412 raft_consensus.cc:2781] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:19:47.370417 8412 raft_consensus.cc:482] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:19:47.370461 8412 raft_consensus.cc:3037] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:47.371479 8412 raft_consensus.cc:504] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.371624 8412 leader_election.cc:288] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab [CANDIDATE]: Term 1 election: Requested vote from peers 2331dfca9ae24769b1aba1eef5e3114a (127.0.98.1:35223), 99153313d5e843eb9d988459ee092574 (127.0.98.2:41125)
I20260924 00:19:47.371789 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7aa4ff744a745b58a8dea6ee798f782" candidate_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2331dfca9ae24769b1aba1eef5e3114a"
I20260924 00:19:47.371820 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7aa4ff744a745b58a8dea6ee798f782" candidate_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99153313d5e843eb9d988459ee092574"
I20260924 00:19:47.371883 8193 raft_consensus.cc:3037] T f7aa4ff744a745b58a8dea6ee798f782 P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:47.371896 8053 raft_consensus.cc:3037] T f7aa4ff744a745b58a8dea6ee798f782 P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:47.372790 8053 raft_consensus.cc:2454] T f7aa4ff744a745b58a8dea6ee798f782 P 2331dfca9ae24769b1aba1eef5e3114a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac83bdbf50345c2b8df30f54cddd7ab in term 1.
I20260924 00:19:47.372843 8193 raft_consensus.cc:2454] T f7aa4ff744a745b58a8dea6ee798f782 P 99153313d5e843eb9d988459ee092574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac83bdbf50345c2b8df30f54cddd7ab in term 1.
I20260924 00:19:47.372959 8265 leader_election.cc:302] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2331dfca9ae24769b1aba1eef5e3114a, fac83bdbf50345c2b8df30f54cddd7ab; no voters:
I20260924 00:19:47.373049 8412 raft_consensus.cc:2781] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:47.373193 8412 raft_consensus.cc:686] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab [term 1 LEADER]: Becoming Leader. State: Replica: fac83bdbf50345c2b8df30f54cddd7ab, State: Running, Role: LEADER
I20260924 00:19:47.373312 8412 consensus_queue.cc:237] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.374327 7896 catalog_manager.cc:5495] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab reported cstate change: term changed from 0 to 1, leader changed from <none> to fac83bdbf50345c2b8df30f54cddd7ab (127.0.98.3). New cstate: current_term: 1 leader_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } health_report { overall_health: HEALTHY } } }
I20260924 00:19:47.407910 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.405631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36784 (local address 127.0.98.3:36929)
0924 00:19:47.405692 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:47.405694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:47.405733 (+ 39us) server_negotiation.cc:407] Connection header received
0924 00:19:47.405816 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:47.405817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:47.405835 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:47.405889 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:47.405994 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.406454 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.407164 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.407341 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.407392 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.407546 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:47.407645 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:47.407708 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:47.407780 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 00:19:47.407798 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 00:19:47.410167 8053 raft_consensus.cc:1261] T f7aa4ff744a745b58a8dea6ee798f782 P 2331dfca9ae24769b1aba1eef5e3114a [term 1 FOLLOWER]: Refusing update from remote peer fac83bdbf50345c2b8df30f54cddd7ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:47.410207 8193 raft_consensus.cc:1261] T f7aa4ff744a745b58a8dea6ee798f782 P 99153313d5e843eb9d988459ee092574 [term 1 FOLLOWER]: Refusing update from remote peer fac83bdbf50345c2b8df30f54cddd7ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:47.410557 8412 consensus_queue.cc:1035] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:47.410714 8417 consensus_queue.cc:1035] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:47.413705 8421 mvcc.cc:205] Tried to move back new op lower bound from 7332696831628214272 to 7332696831482003456. Current Snapshot: MvccSnapshot[applied={T|T < 7332696831628214272}]
I20260924 00:19:47.414188 8422 mvcc.cc:205] Tried to move back new op lower bound from 7332696831628214272 to 7332696831482003456. Current Snapshot: MvccSnapshot[applied={T|T < 7332696831628214272}]
I20260924 00:19:47.415786 7895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:47.415885 7895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:47.416745 7895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be: Sending DeleteTablet for 3 replicas of tablet f7aa4ff744a745b58a8dea6ee798f782
I20260924 00:19:47.417109 8312 tablet_service.cc:1514] Processing DeleteTablet for tablet f7aa4ff744a745b58a8dea6ee798f782 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36780
I20260924 00:19:47.417279 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet f7aa4ff744a745b58a8dea6ee798f782 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59752
I20260924 00:19:47.417321 8033 tablet_service.cc:1514] Processing DeleteTablet for tablet f7aa4ff744a745b58a8dea6ee798f782 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42322
I20260924 00:19:47.417531 8429 tablet_replica.cc:330] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab: stopping tablet replica
I20260924 00:19:47.417652 8429 raft_consensus.cc:2229] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:19:47.417737 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 00:19:47.417824 8429 raft_consensus.cc:2258] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 00:19:47.417914 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:19:47.418203 8431 tablet_replica.cc:330] T f7aa4ff744a745b58a8dea6ee798f782 P 2331dfca9ae24769b1aba1eef5e3114a: stopping tablet replica
I20260924 00:19:47.418313 8429 ts_tablet_manager.cc:1905] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:47.419226 8173 tablet_service.cc:1467] Processing CreateTablet for tablet 909448f77eca402cbb8bc06ed216298c (DEFAULT_TABLE table=test-table [id=8889e71381fd4f058e3e3db59a08bab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:47.419226 8312 tablet_service.cc:1467] Processing CreateTablet for tablet 909448f77eca402cbb8bc06ed216298c (DEFAULT_TABLE table=test-table [id=8889e71381fd4f058e3e3db59a08bab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:47.419337 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 909448f77eca402cbb8bc06ed216298c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:47.419376 8312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 909448f77eca402cbb8bc06ed216298c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:47.420225 8429 ts_tablet_manager.cc:1918] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:19:47.420285 8429 log.cc:1196] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/f7aa4ff744a745b58a8dea6ee798f782
I20260924 00:19:47.420547 8429 ts_tablet_manager.cc:1939] T f7aa4ff744a745b58a8dea6ee798f782 P fac83bdbf50345c2b8df30f54cddd7ab: Deleting consensus metadata
I20260924 00:19:47.420666 8407 tablet_bootstrap.cc:492] T 909448f77eca402cbb8bc06ed216298c P fac83bdbf50345c2b8df30f54cddd7ab: Bootstrap starting.
I20260924 00:19:47.420679 8411 tablet_bootstrap.cc:492] T 909448f77eca402cbb8bc06ed216298c P 99153313d5e843eb9d988459ee092574: Bootstrap starting.
I20260924 00:19:47.421103 8411 tablet_bootstrap.cc:654] T 909448f77eca402cbb8bc06ed216298c P 99153313d5e843eb9d988459ee092574: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:47.421103 8407 tablet_bootstrap.cc:654] T 909448f77eca402cbb8bc06ed216298c P fac83bdbf50345c2b8df30f54cddd7ab: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:47.421576 7882 catalog_manager.cc:4841] TS fac83bdbf50345c2b8df30f54cddd7ab (127.0.98.3:36929): tablet f7aa4ff744a745b58a8dea6ee798f782 (table test-table [id=bdb44f00732e418d8f23875e2f46a7ce]) successfully deleted
I20260924 00:19:47.421730 8411 tablet_bootstrap.cc:492] T 909448f77eca402cbb8bc06ed216298c P 99153313d5e843eb9d988459ee092574: No bootstrap required, opened a new log
I20260924 00:19:47.421783 8411 ts_tablet_manager.cc:1397] T 909448f77eca402cbb8bc06ed216298c P 99153313d5e843eb9d988459ee092574: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:47.421931 8411 raft_consensus.cc:348] T 909448f77eca402cbb8bc06ed216298c P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.421821 8033 tablet_service.cc:1467] Processing CreateTablet for tablet 909448f77eca402cbb8bc06ed216298c (DEFAULT_TABLE table=test-table [id=8889e71381fd4f058e3e3db59a08bab1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:47.422013 8411 raft_consensus.cc:374] T 909448f77eca402cbb8bc06ed216298c P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:47.422044 8411 raft_consensus.cc:729] T 909448f77eca402cbb8bc06ed216298c P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99153313d5e843eb9d988459ee092574, State: Initialized, Role: FOLLOWER
I20260924 00:19:47.422076 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 909448f77eca402cbb8bc06ed216298c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:47.422089 8411 consensus_queue.cc:260] T 909448f77eca402cbb8bc06ed216298c P 99153313d5e843eb9d988459ee092574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.422175 8411 ts_tablet_manager.cc:1428] T 909448f77eca402cbb8bc06ed216298c P 99153313d5e843eb9d988459ee092574: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:47.422588 8407 tablet_bootstrap.cc:492] T 909448f77eca402cbb8bc06ed216298c P fac83bdbf50345c2b8df30f54cddd7ab: No bootstrap required, opened a new log
I20260924 00:19:47.422636 8407 ts_tablet_manager.cc:1397] T 909448f77eca402cbb8bc06ed216298c P fac83bdbf50345c2b8df30f54cddd7ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:19:47.422762 8407 raft_consensus.cc:348] T 909448f77eca402cbb8bc06ed216298c P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.422816 8407 raft_consensus.cc:374] T 909448f77eca402cbb8bc06ed216298c P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:47.422837 8407 raft_consensus.cc:729] T 909448f77eca402cbb8bc06ed216298c P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac83bdbf50345c2b8df30f54cddd7ab, State: Initialized, Role: FOLLOWER
I20260924 00:19:47.422875 8407 consensus_queue.cc:260] T 909448f77eca402cbb8bc06ed216298c P fac83bdbf50345c2b8df30f54cddd7ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.422957 8407 ts_tablet_manager.cc:1428] T 909448f77eca402cbb8bc06ed216298c P fac83bdbf50345c2b8df30f54cddd7ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:47.423530 8408 tablet_bootstrap.cc:492] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a: Bootstrap starting.
I20260924 00:19:47.423614 8431 raft_consensus.cc:2229] T f7aa4ff744a745b58a8dea6ee798f782 P 2331dfca9ae24769b1aba1eef5e3114a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:47.423797 8431 raft_consensus.cc:2258] T f7aa4ff744a745b58a8dea6ee798f782 P 2331dfca9ae24769b1aba1eef5e3114a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:47.423913 8408 tablet_bootstrap.cc:654] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:47.423971 8430 tablet_replica.cc:330] T f7aa4ff744a745b58a8dea6ee798f782 P 99153313d5e843eb9d988459ee092574: stopping tablet replica
I20260924 00:19:47.424062 8430 raft_consensus.cc:2229] T f7aa4ff744a745b58a8dea6ee798f782 P 99153313d5e843eb9d988459ee092574 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:47.424124 8430 raft_consensus.cc:2258] T f7aa4ff744a745b58a8dea6ee798f782 P 99153313d5e843eb9d988459ee092574 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:47.424558 8431 ts_tablet_manager.cc:1905] T f7aa4ff744a745b58a8dea6ee798f782 P 2331dfca9ae24769b1aba1eef5e3114a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:47.424605 8408 tablet_bootstrap.cc:492] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a: No bootstrap required, opened a new log
I20260924 00:19:47.424638 8408 ts_tablet_manager.cc:1397] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:47.424695 8430 ts_tablet_manager.cc:1905] T f7aa4ff744a745b58a8dea6ee798f782 P 99153313d5e843eb9d988459ee092574: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:47.424788 8408 raft_consensus.cc:348] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.424851 8408 raft_consensus.cc:374] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:47.424870 8408 raft_consensus.cc:729] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2331dfca9ae24769b1aba1eef5e3114a, State: Initialized, Role: FOLLOWER
I20260924 00:19:47.424930 8408 consensus_queue.cc:260] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.425019 8408 ts_tablet_manager.cc:1428] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:47.426080 8431 ts_tablet_manager.cc:1918] T f7aa4ff744a745b58a8dea6ee798f782 P 2331dfca9ae24769b1aba1eef5e3114a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:19:47.426155 8431 log.cc:1196] T f7aa4ff744a745b58a8dea6ee798f782 P 2331dfca9ae24769b1aba1eef5e3114a: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-0/wal/wals/f7aa4ff744a745b58a8dea6ee798f782
I20260924 00:19:47.426189 8430 ts_tablet_manager.cc:1918] T f7aa4ff744a745b58a8dea6ee798f782 P 99153313d5e843eb9d988459ee092574: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:19:47.426254 8430 log.cc:1196] T f7aa4ff744a745b58a8dea6ee798f782 P 99153313d5e843eb9d988459ee092574: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-1/wal/wals/f7aa4ff744a745b58a8dea6ee798f782
I20260924 00:19:47.426690 8430 ts_tablet_manager.cc:1939] T f7aa4ff744a745b58a8dea6ee798f782 P 99153313d5e843eb9d988459ee092574: Deleting consensus metadata
I20260924 00:19:47.426749 8431 ts_tablet_manager.cc:1939] T f7aa4ff744a745b58a8dea6ee798f782 P 2331dfca9ae24769b1aba1eef5e3114a: Deleting consensus metadata
I20260924 00:19:47.427142 7881 catalog_manager.cc:4841] TS 99153313d5e843eb9d988459ee092574 (127.0.98.2:41125): tablet f7aa4ff744a745b58a8dea6ee798f782 (table test-table [id=bdb44f00732e418d8f23875e2f46a7ce]) successfully deleted
I20260924 00:19:47.427218 7881 catalog_manager.cc:4841] TS 2331dfca9ae24769b1aba1eef5e3114a (127.0.98.1:35223): tablet f7aa4ff744a745b58a8dea6ee798f782 (table test-table [id=bdb44f00732e418d8f23875e2f46a7ce]) successfully deleted
W20260924 00:19:47.667717 8380 tablet.cc:2369] T 909448f77eca402cbb8bc06ed216298c P fac83bdbf50345c2b8df30f54cddd7ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:19:47.708765 8414 raft_consensus.cc:482] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:19:47.708875 8414 raft_consensus.cc:504] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.709296 8414 leader_election.cc:288] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99153313d5e843eb9d988459ee092574 (127.0.98.2:41125), fac83bdbf50345c2b8df30f54cddd7ab (127.0.98.3:36929)
I20260924 00:19:47.712726 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.709503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36929 (local address 127.0.98.1:47183)
0924 00:19:47.709693 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:47.709704 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:47.709782 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:47.709970 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:47.709971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:47.709973 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:47.710079 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.710081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.710737 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.710738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.711772 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.711775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.712387 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.712388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.712493 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.712498 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:19:47.712568 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 00:19:47.712619 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":97,"threads_started":1}
I20260924 00:19:47.712749 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.709638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47183 (local address 127.0.98.3:36929)
0924 00:19:47.709688 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:47.709690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:47.709781 (+ 91us) server_negotiation.cc:407] Connection header received
0924 00:19:47.709856 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:47.709856 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:47.709889 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:47.709945 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:47.710155 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.710643 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.711853 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.712303 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.712372 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.712382 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:47.712626 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 00:19:47.712671 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 00:19:47.712924 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.709490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41125 (local address 127.0.98.1:49333)
0924 00:19:47.709644 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:47.709656 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:47.709732 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:47.709916 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:47.709917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:47.709919 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:19:47.710011 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.710013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.710787 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.710787 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.711800 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.711803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.712645 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.712647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.712714 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.712719 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:19:47.712784 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 00:19:47.712825 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":50,"thread_start_us":64,"threads_started":1}
I20260924 00:19:47.712970 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.709570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49333 (local address 127.0.98.2:41125)
0924 00:19:47.709625 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:47.709628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:47.709741 (+ 113us) server_negotiation.cc:407] Connection header received
0924 00:19:47.709820 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:47.709822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:47.709833 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:47.709909 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:19:47.710084 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.710663 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.711923 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.712537 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.712609 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.712619 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:47.712811 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 00:19:47.712866 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:19:47.713299 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "909448f77eca402cbb8bc06ed216298c" candidate_uuid: "2331dfca9ae24769b1aba1eef5e3114a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" is_pre_election: true
I20260924 00:19:47.713478 8333 raft_consensus.cc:2454] T 909448f77eca402cbb8bc06ed216298c P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2331dfca9ae24769b1aba1eef5e3114a in term 0.
I20260924 00:19:47.713557 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "909448f77eca402cbb8bc06ed216298c" candidate_uuid: "2331dfca9ae24769b1aba1eef5e3114a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99153313d5e843eb9d988459ee092574" is_pre_election: true
I20260924 00:19:47.713642 8193 raft_consensus.cc:2454] T 909448f77eca402cbb8bc06ed216298c P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2331dfca9ae24769b1aba1eef5e3114a in term 0.
I20260924 00:19:47.713663 7986 leader_election.cc:302] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2331dfca9ae24769b1aba1eef5e3114a, fac83bdbf50345c2b8df30f54cddd7ab; no voters:
I20260924 00:19:47.713800 8414 raft_consensus.cc:2781] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:19:47.713838 8414 raft_consensus.cc:482] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:19:47.713853 8414 raft_consensus.cc:3037] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:47.714533 8414 raft_consensus.cc:504] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.714691 8414 leader_election.cc:288] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a [CANDIDATE]: Term 1 election: Requested vote from peers 99153313d5e843eb9d988459ee092574 (127.0.98.2:41125), fac83bdbf50345c2b8df30f54cddd7ab (127.0.98.3:36929)
I20260924 00:19:47.714836 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "909448f77eca402cbb8bc06ed216298c" candidate_uuid: "2331dfca9ae24769b1aba1eef5e3114a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fac83bdbf50345c2b8df30f54cddd7ab"
I20260924 00:19:47.714895 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "909448f77eca402cbb8bc06ed216298c" candidate_uuid: "2331dfca9ae24769b1aba1eef5e3114a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99153313d5e843eb9d988459ee092574"
I20260924 00:19:47.714950 8193 raft_consensus.cc:3037] T 909448f77eca402cbb8bc06ed216298c P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:47.714944 8333 raft_consensus.cc:3037] T 909448f77eca402cbb8bc06ed216298c P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:47.715705 8193 raft_consensus.cc:2454] T 909448f77eca402cbb8bc06ed216298c P 99153313d5e843eb9d988459ee092574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2331dfca9ae24769b1aba1eef5e3114a in term 1.
I20260924 00:19:47.715862 8333 raft_consensus.cc:2454] T 909448f77eca402cbb8bc06ed216298c P fac83bdbf50345c2b8df30f54cddd7ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2331dfca9ae24769b1aba1eef5e3114a in term 1.
I20260924 00:19:47.715927 7985 leader_election.cc:302] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2331dfca9ae24769b1aba1eef5e3114a, 99153313d5e843eb9d988459ee092574; no voters:
I20260924 00:19:47.716046 8414 raft_consensus.cc:2781] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:47.716279 8414 raft_consensus.cc:686] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a [term 1 LEADER]: Becoming Leader. State: Replica: 2331dfca9ae24769b1aba1eef5e3114a, State: Running, Role: LEADER
I20260924 00:19:47.716370 8414 consensus_queue.cc:237] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.717168 7895 catalog_manager.cc:5495] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2331dfca9ae24769b1aba1eef5e3114a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2331dfca9ae24769b1aba1eef5e3114a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } health_report { overall_health: UNKNOWN } } }
Sep 24 00:19:47 dist-test-slave-777x krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209186, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:19:47.800544 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.793148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32812 (local address 127.0.98.62:43107)
0924 00:19:47.793383 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:47.793385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:47.793396 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:19:47.793420 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:47.793421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:47.793432 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:47.793509 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:47.794006 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.794527 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.795221 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.795402 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.795453 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.797950 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:47.797951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:47.797952 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:47.797958 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:47.799297 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:47.799743 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:47.799744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:47.799744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:47.799806 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:47.800035 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:47.800035 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:47.800035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:47.800139 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:47.800252 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:47.800399 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 00:19:47.800471 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":102,"threads_started":1}
I20260924 00:19:47.800671 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.793061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43107 (local address 127.0.0.1:32812)
0924 00:19:47.793250 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:47.793264 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:47.793341 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:47.793526 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:47.793527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:47.793846 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:47.793913 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.793915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.794624 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.794625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.795147 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.795149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.795490 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.795492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.795542 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.796279 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:47.796285 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:47.797853 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:47.799381 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:47.799388 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:47.799390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:47.799672 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:47.799895 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:47.799897 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:47.799897 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:47.799967 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:47.800245 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:47.800257 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:47.800305 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 00:19:47.800370 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 00:19:47.800535 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":102,"threads_started":1}
I20260924 00:19:47.802971 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32812:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 00:19:47.803184 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:19:47.804865 8033 tablet_service.cc:1467] Processing CreateTablet for tablet 1354ab6e730440d8bd89ee1a2cb0828a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c528b8e6cf284112a2acc0d460821acb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:19:47.804876 8173 tablet_service.cc:1467] Processing CreateTablet for tablet 1354ab6e730440d8bd89ee1a2cb0828a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c528b8e6cf284112a2acc0d460821acb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:19:47.805030 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1354ab6e730440d8bd89ee1a2cb0828a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:47.805075 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1354ab6e730440d8bd89ee1a2cb0828a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:47.806264 8411 tablet_bootstrap.cc:492] T 1354ab6e730440d8bd89ee1a2cb0828a P 99153313d5e843eb9d988459ee092574: Bootstrap starting.
I20260924 00:19:47.806464 8408 tablet_bootstrap.cc:492] T 1354ab6e730440d8bd89ee1a2cb0828a P 2331dfca9ae24769b1aba1eef5e3114a: Bootstrap starting.
I20260924 00:19:47.806589 8312 tablet_service.cc:1467] Processing CreateTablet for tablet 1354ab6e730440d8bd89ee1a2cb0828a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c528b8e6cf284112a2acc0d460821acb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 00:19:47.806689 8411 tablet_bootstrap.cc:654] T 1354ab6e730440d8bd89ee1a2cb0828a P 99153313d5e843eb9d988459ee092574: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:47.806713 8312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1354ab6e730440d8bd89ee1a2cb0828a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:47.806850 8408 tablet_bootstrap.cc:654] T 1354ab6e730440d8bd89ee1a2cb0828a P 2331dfca9ae24769b1aba1eef5e3114a: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:47.807336 8411 tablet_bootstrap.cc:492] T 1354ab6e730440d8bd89ee1a2cb0828a P 99153313d5e843eb9d988459ee092574: No bootstrap required, opened a new log
I20260924 00:19:47.807394 8411 ts_tablet_manager.cc:1397] T 1354ab6e730440d8bd89ee1a2cb0828a P 99153313d5e843eb9d988459ee092574: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:47.807543 8411 raft_consensus.cc:348] T 1354ab6e730440d8bd89ee1a2cb0828a P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } }
I20260924 00:19:47.807641 8411 raft_consensus.cc:374] T 1354ab6e730440d8bd89ee1a2cb0828a P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:47.807664 8411 raft_consensus.cc:729] T 1354ab6e730440d8bd89ee1a2cb0828a P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99153313d5e843eb9d988459ee092574, State: Initialized, Role: FOLLOWER
I20260924 00:19:47.807709 8411 consensus_queue.cc:260] T 1354ab6e730440d8bd89ee1a2cb0828a P 99153313d5e843eb9d988459ee092574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } }
I20260924 00:19:47.807799 8411 ts_tablet_manager.cc:1428] T 1354ab6e730440d8bd89ee1a2cb0828a P 99153313d5e843eb9d988459ee092574: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 00:19:47.807910 8415 tablet_replica.cc:439] T 1354ab6e730440d8bd89ee1a2cb0828a P 99153313d5e843eb9d988459ee092574: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } }
I20260924 00:19:47.807999 8415 tablet_replica.cc:442] T 1354ab6e730440d8bd89ee1a2cb0828a P 99153313d5e843eb9d988459ee092574: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:19:47.808398 8407 tablet_bootstrap.cc:492] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab: Bootstrap starting.
I20260924 00:19:47.808460 8408 tablet_bootstrap.cc:492] T 1354ab6e730440d8bd89ee1a2cb0828a P 2331dfca9ae24769b1aba1eef5e3114a: No bootstrap required, opened a new log
I20260924 00:19:47.808508 8408 ts_tablet_manager.cc:1397] T 1354ab6e730440d8bd89ee1a2cb0828a P 2331dfca9ae24769b1aba1eef5e3114a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 00:19:47.808667 8408 raft_consensus.cc:348] T 1354ab6e730440d8bd89ee1a2cb0828a P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } }
I20260924 00:19:47.808763 8408 raft_consensus.cc:374] T 1354ab6e730440d8bd89ee1a2cb0828a P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:47.808789 8408 raft_consensus.cc:729] T 1354ab6e730440d8bd89ee1a2cb0828a P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2331dfca9ae24769b1aba1eef5e3114a, State: Initialized, Role: FOLLOWER
I20260924 00:19:47.808818 8407 tablet_bootstrap.cc:654] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:47.808844 8408 consensus_queue.cc:260] T 1354ab6e730440d8bd89ee1a2cb0828a P 2331dfca9ae24769b1aba1eef5e3114a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } }
I20260924 00:19:47.808950 8408 ts_tablet_manager.cc:1428] T 1354ab6e730440d8bd89ee1a2cb0828a P 2331dfca9ae24769b1aba1eef5e3114a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 00:19:47.809094 8414 tablet_replica.cc:439] T 1354ab6e730440d8bd89ee1a2cb0828a P 2331dfca9ae24769b1aba1eef5e3114a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } }
I20260924 00:19:47.809216 8414 tablet_replica.cc:442] T 1354ab6e730440d8bd89ee1a2cb0828a P 2331dfca9ae24769b1aba1eef5e3114a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:19:47.809546 8407 tablet_bootstrap.cc:492] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab: No bootstrap required, opened a new log
I20260924 00:19:47.809635 8407 ts_tablet_manager.cc:1397] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:47.809772 8407 raft_consensus.cc:348] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } }
I20260924 00:19:47.809862 8407 raft_consensus.cc:374] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:47.809900 8407 raft_consensus.cc:729] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac83bdbf50345c2b8df30f54cddd7ab, State: Initialized, Role: FOLLOWER
I20260924 00:19:47.809965 8407 consensus_queue.cc:260] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } }
I20260924 00:19:47.810103 8407 ts_tablet_manager.cc:1428] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 00:19:47.810151 8417 tablet_replica.cc:439] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } }
I20260924 00:19:47.810257 8417 tablet_replica.cc:442] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:19:47.817451 8417 raft_consensus.cc:482] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:19:47.817552 8417 raft_consensus.cc:504] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } }
I20260924 00:19:47.817740 8417 leader_election.cc:288] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2331dfca9ae24769b1aba1eef5e3114a (127.0.98.1:35223), 99153313d5e843eb9d988459ee092574 (127.0.98.2:41125)
I20260924 00:19:47.817943 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1354ab6e730440d8bd89ee1a2cb0828a" candidate_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2331dfca9ae24769b1aba1eef5e3114a" is_pre_election: true
I20260924 00:19:47.817966 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1354ab6e730440d8bd89ee1a2cb0828a" candidate_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99153313d5e843eb9d988459ee092574" is_pre_election: true
I20260924 00:19:47.818048 8053 raft_consensus.cc:2454] T 1354ab6e730440d8bd89ee1a2cb0828a P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac83bdbf50345c2b8df30f54cddd7ab in term 0.
I20260924 00:19:47.818058 8193 raft_consensus.cc:2454] T 1354ab6e730440d8bd89ee1a2cb0828a P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac83bdbf50345c2b8df30f54cddd7ab in term 0.
I20260924 00:19:47.818214 8265 leader_election.cc:302] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2331dfca9ae24769b1aba1eef5e3114a, fac83bdbf50345c2b8df30f54cddd7ab; no voters:
I20260924 00:19:47.818318 8417 raft_consensus.cc:2781] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:19:47.818359 8417 raft_consensus.cc:482] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:19:47.818380 8417 raft_consensus.cc:3037] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:47.819145 8417 raft_consensus.cc:504] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } }
I20260924 00:19:47.819315 8417 leader_election.cc:288] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab [CANDIDATE]: Term 1 election: Requested vote from peers 2331dfca9ae24769b1aba1eef5e3114a (127.0.98.1:35223), 99153313d5e843eb9d988459ee092574 (127.0.98.2:41125)
I20260924 00:19:47.819507 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1354ab6e730440d8bd89ee1a2cb0828a" candidate_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99153313d5e843eb9d988459ee092574"
I20260924 00:19:47.819554 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1354ab6e730440d8bd89ee1a2cb0828a" candidate_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2331dfca9ae24769b1aba1eef5e3114a"
I20260924 00:19:47.819593 8193 raft_consensus.cc:3037] T 1354ab6e730440d8bd89ee1a2cb0828a P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:47.819615 8053 raft_consensus.cc:3037] T 1354ab6e730440d8bd89ee1a2cb0828a P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:47.820341 8193 raft_consensus.cc:2454] T 1354ab6e730440d8bd89ee1a2cb0828a P 99153313d5e843eb9d988459ee092574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac83bdbf50345c2b8df30f54cddd7ab in term 1.
I20260924 00:19:47.820363 8053 raft_consensus.cc:2454] T 1354ab6e730440d8bd89ee1a2cb0828a P 2331dfca9ae24769b1aba1eef5e3114a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac83bdbf50345c2b8df30f54cddd7ab in term 1.
I20260924 00:19:47.820562 8265 leader_election.cc:302] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99153313d5e843eb9d988459ee092574, fac83bdbf50345c2b8df30f54cddd7ab; no voters:
I20260924 00:19:47.820660 8417 raft_consensus.cc:2781] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:47.820713 8417 raft_consensus.cc:686] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab [term 1 LEADER]: Becoming Leader. State: Replica: fac83bdbf50345c2b8df30f54cddd7ab, State: Running, Role: LEADER
I20260924 00:19:47.820785 8417 consensus_queue.cc:237] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } }
I20260924 00:19:47.821058 8412 tablet_replica.cc:439] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab: TxnStatusTablet state changed. Reason: New leader fac83bdbf50345c2b8df30f54cddd7ab. Latest consensus state: current_term: 1 leader_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } }
I20260924 00:19:47.821163 8412 tablet_replica.cc:442] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:19:47.821350 7895 catalog_manager.cc:5495] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab reported cstate change: term changed from 0 to 1, leader changed from <none> to fac83bdbf50345c2b8df30f54cddd7ab (127.0.98.3). New cstate: current_term: 1 leader_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } health_report { overall_health: UNKNOWN } } }
I20260924 00:19:47.821671 8450 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 00:19:47.821805 8450 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 00:19:47.826654 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.824292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36800 (local address 127.0.98.3:36929)
0924 00:19:47.824336 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:47.824338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:47.824347 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:19:47.824427 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:47.824428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:47.824441 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:47.824555 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:47.824693 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.825176 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.825926 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.826134 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.826207 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.826377 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:47.826410 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:47.826494 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:47.826561 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 00:19:47.826584 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:19:47.826664 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.824235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36929 (local address 127.0.0.1:36800)
0924 00:19:47.824293 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:47.824304 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:47.824355 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:47.824563 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:47.824564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:47.824565 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:19:47.824613 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.824616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.825258 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.825259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.825840 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.825841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.826206 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.826207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.826260 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.826284 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:19:47.826498 (+ 214us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:19:47.826507 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 00:19:47.826548 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 00:19:47.826599 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 00:19:47.827642 8193 raft_consensus.cc:1261] T 1354ab6e730440d8bd89ee1a2cb0828a P 99153313d5e843eb9d988459ee092574 [term 1 FOLLOWER]: Refusing update from remote peer fac83bdbf50345c2b8df30f54cddd7ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:47.827682 8053 raft_consensus.cc:1261] T 1354ab6e730440d8bd89ee1a2cb0828a P 2331dfca9ae24769b1aba1eef5e3114a [term 1 FOLLOWER]: Refusing update from remote peer fac83bdbf50345c2b8df30f54cddd7ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:47.827862 8417 consensus_queue.cc:1035] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:47.827935 8412 consensus_queue.cc:1035] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:47.828456 8415 tablet_replica.cc:439] T 1354ab6e730440d8bd89ee1a2cb0828a P 99153313d5e843eb9d988459ee092574: TxnStatusTablet state changed. Reason: New leader fac83bdbf50345c2b8df30f54cddd7ab. Latest consensus state: current_term: 1 leader_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } }
I20260924 00:19:47.828613 8415 tablet_replica.cc:442] T 1354ab6e730440d8bd89ee1a2cb0828a P 99153313d5e843eb9d988459ee092574: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:19:47.828891 8414 tablet_replica.cc:439] T 1354ab6e730440d8bd89ee1a2cb0828a P 2331dfca9ae24769b1aba1eef5e3114a: TxnStatusTablet state changed. Reason: New leader fac83bdbf50345c2b8df30f54cddd7ab. Latest consensus state: current_term: 1 leader_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } }
I20260924 00:19:47.828991 8414 tablet_replica.cc:442] T 1354ab6e730440d8bd89ee1a2cb0828a P 2331dfca9ae24769b1aba1eef5e3114a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:19:47.829142 8417 tablet_replica.cc:439] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } }
I20260924 00:19:47.829239 8417 tablet_replica.cc:442] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:19:47.829705 8420 mvcc.cc:205] Tried to move back new op lower bound from 7332696833340248064 to 7332696833314275328. Current Snapshot: MvccSnapshot[applied={T|T < 7332696833340248064 or (T in {7332696833340248064})}]
I20260924 00:19:47.830194 8412 tablet_replica.cc:439] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } }
I20260924 00:19:47.830282 8412 tablet_replica.cc:442] T 1354ab6e730440d8bd89ee1a2cb0828a P fac83bdbf50345c2b8df30f54cddd7ab: This TxnStatusTablet replica's current role is: LEADER
I20260924 00:19:47.830242 8415 tablet_replica.cc:439] T 1354ab6e730440d8bd89ee1a2cb0828a P 99153313d5e843eb9d988459ee092574: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } }
I20260924 00:19:47.830336 8415 tablet_replica.cc:442] T 1354ab6e730440d8bd89ee1a2cb0828a P 99153313d5e843eb9d988459ee092574: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:19:47.830605 8414 tablet_replica.cc:439] T 1354ab6e730440d8bd89ee1a2cb0828a P 2331dfca9ae24769b1aba1eef5e3114a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } }
I20260924 00:19:47.830699 8414 tablet_replica.cc:442] T 1354ab6e730440d8bd89ee1a2cb0828a P 2331dfca9ae24769b1aba1eef5e3114a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 00:19:47.833791 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.831326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42338 (local address 127.0.98.1:35223)
0924 00:19:47.831365 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:47.831367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:47.831388 (+ 21us) server_negotiation.cc:407] Connection header received
0924 00:19:47.831422 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:47.831423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:47.831444 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:47.831521 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:47.831660 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.832148 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.833001 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.833205 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.833289 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.833461 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:47.833567 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:47.833653 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:47.833692 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 00:19:47.833713 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 00:19:47.839275 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.836372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53965 (local address 127.0.98.3:36929)
0924 00:19:47.836417 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:47.836419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:47.836614 (+ 195us) server_negotiation.cc:407] Connection header received
0924 00:19:47.836674 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:47.836675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:47.836684 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:47.836725 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:47.836869 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.837304 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.838558 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.838740 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.838795 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.839026 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:47.839057 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:47.839115 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:47.839191 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 00:19:47.839213 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:19:47.839324 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.836292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36929 (local address 127.0.98.1:53965)
0924 00:19:47.836536 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:47.836548 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:47.836612 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:47.836738 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:47.836739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:47.836740 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:19:47.836811 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.836812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.837424 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.837425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.838466 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.838469 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.838818 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.838819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.838888 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.838905 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:19:47.839132 (+ 227us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:19:47.839134 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:19:47.839180 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 00:19:47.839227 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":168,"threads_started":1}
I20260924 00:19:47.843138 8333 raft_consensus.cc:1261] T 909448f77eca402cbb8bc06ed216298c P fac83bdbf50345c2b8df30f54cddd7ab [term 1 FOLLOWER]: Refusing update from remote peer 2331dfca9ae24769b1aba1eef5e3114a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:47.843127 8192 raft_consensus.cc:1261] T 909448f77eca402cbb8bc06ed216298c P 99153313d5e843eb9d988459ee092574 [term 1 FOLLOWER]: Refusing update from remote peer 2331dfca9ae24769b1aba1eef5e3114a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:47.843369 8414 consensus_queue.cc:1035] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:47.843503 8437 consensus_queue.cc:1035] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:47.855835 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.852767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35223 (local address 127.0.98.3:39123)
0924 00:19:47.853020 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:47.853032 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:47.853517 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:47.853696 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:47.853697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:47.853698 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 00:19:47.853750 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.853752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.854441 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.854442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.855005 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:47.855007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.855389 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.855390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:47.855459 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.855473 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 00:19:47.855661 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 00:19:47.855663 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:19:47.855713 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 00:19:47.855756 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":98,"threads_started":1}
I20260924 00:19:47.855835 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.852970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39123 (local address 127.0.98.1:35223)
0924 00:19:47.853013 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:47.853015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:47.853085 (+ 70us) server_negotiation.cc:407] Connection header received
0924 00:19:47.853585 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:47.853586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:47.853599 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:47.853647 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:47.853831 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.854363 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.855093 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.855298 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.855388 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.855568 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:47.855601 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:47.855651 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:47.855718 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 00:19:47.855736 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 00:19:47.865741 7896 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:47.865861 7896 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:47.866523 7896 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be: Sending DeleteTablet for 3 replicas of tablet 909448f77eca402cbb8bc06ed216298c
I20260924 00:19:47.866771 8033 tablet_service.cc:1514] Processing DeleteTablet for tablet 909448f77eca402cbb8bc06ed216298c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42322
I20260924 00:19:47.866861 8431 tablet_replica.cc:330] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a: stopping tablet replica
I20260924 00:19:47.866864 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet 909448f77eca402cbb8bc06ed216298c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59752
I20260924 00:19:47.866912 8431 raft_consensus.cc:2229] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:19:47.866989 8431 raft_consensus.cc:2258] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:47.866979 8312 tablet_service.cc:1514] Processing DeleteTablet for tablet 909448f77eca402cbb8bc06ed216298c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36780
I20260924 00:19:47.866993 8430 tablet_replica.cc:330] T 909448f77eca402cbb8bc06ed216298c P 99153313d5e843eb9d988459ee092574: stopping tablet replica
I20260924 00:19:47.867050 8430 raft_consensus.cc:2229] T 909448f77eca402cbb8bc06ed216298c P 99153313d5e843eb9d988459ee092574 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:47.867066 8429 tablet_replica.cc:330] T 909448f77eca402cbb8bc06ed216298c P fac83bdbf50345c2b8df30f54cddd7ab: stopping tablet replica
I20260924 00:19:47.867085 8430 raft_consensus.cc:2258] T 909448f77eca402cbb8bc06ed216298c P 99153313d5e843eb9d988459ee092574 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:47.867122 8429 raft_consensus.cc:2229] T 909448f77eca402cbb8bc06ed216298c P fac83bdbf50345c2b8df30f54cddd7ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:47.867166 8429 raft_consensus.cc:2258] T 909448f77eca402cbb8bc06ed216298c P fac83bdbf50345c2b8df30f54cddd7ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:47.867364 8431 ts_tablet_manager.cc:1905] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:47.867486 8429 ts_tablet_manager.cc:1905] T 909448f77eca402cbb8bc06ed216298c P fac83bdbf50345c2b8df30f54cddd7ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:47.868366 8430 ts_tablet_manager.cc:1905] T 909448f77eca402cbb8bc06ed216298c P 99153313d5e843eb9d988459ee092574: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:47.869472 8429 ts_tablet_manager.cc:1918] T 909448f77eca402cbb8bc06ed216298c P fac83bdbf50345c2b8df30f54cddd7ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:19:47.869521 8429 log.cc:1196] T 909448f77eca402cbb8bc06ed216298c P fac83bdbf50345c2b8df30f54cddd7ab: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/909448f77eca402cbb8bc06ed216298c
I20260924 00:19:47.869732 8429 ts_tablet_manager.cc:1939] T 909448f77eca402cbb8bc06ed216298c P fac83bdbf50345c2b8df30f54cddd7ab: Deleting consensus metadata
I20260924 00:19:47.870111 8430 ts_tablet_manager.cc:1918] T 909448f77eca402cbb8bc06ed216298c P 99153313d5e843eb9d988459ee092574: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:19:47.870164 8430 log.cc:1196] T 909448f77eca402cbb8bc06ed216298c P 99153313d5e843eb9d988459ee092574: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-1/wal/wals/909448f77eca402cbb8bc06ed216298c
I20260924 00:19:47.870257 7882 catalog_manager.cc:4841] TS fac83bdbf50345c2b8df30f54cddd7ab (127.0.98.3:36929): tablet 909448f77eca402cbb8bc06ed216298c (table test-table [id=8889e71381fd4f058e3e3db59a08bab1]) successfully deleted
I20260924 00:19:47.870374 8430 ts_tablet_manager.cc:1939] T 909448f77eca402cbb8bc06ed216298c P 99153313d5e843eb9d988459ee092574: Deleting consensus metadata
I20260924 00:19:47.870685 7881 catalog_manager.cc:4841] TS 99153313d5e843eb9d988459ee092574 (127.0.98.2:41125): tablet 909448f77eca402cbb8bc06ed216298c (table test-table [id=8889e71381fd4f058e3e3db59a08bab1]) successfully deleted
I20260924 00:19:47.871315 8431 ts_tablet_manager.cc:1918] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:19:47.871390 8431 log.cc:1196] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-0/wal/wals/909448f77eca402cbb8bc06ed216298c
I20260924 00:19:47.871584 8431 ts_tablet_manager.cc:1939] T 909448f77eca402cbb8bc06ed216298c P 2331dfca9ae24769b1aba1eef5e3114a: Deleting consensus metadata
I20260924 00:19:47.871954 7881 catalog_manager.cc:4841] TS 2331dfca9ae24769b1aba1eef5e3114a (127.0.98.1:35223): tablet 909448f77eca402cbb8bc06ed216298c (table test-table [id=8889e71381fd4f058e3e3db59a08bab1]) successfully deleted
Sep 24 00:19:47 dist-test-slave-777x krb5kdc[7851](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 00:19:47 dist-test-slave-777x krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:47 dist-test-slave-777x krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 00:19:47.903499 8445 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:47.897896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32826 (local address 127.0.98.62:43107)
0924 00:19:47.897947 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:47.897949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:47.898153 (+ 204us) server_negotiation.cc:407] Connection header received
0924 00:19:47.898241 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:47.898242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:47.898257 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:47.898323 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:47.898906 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.899423 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.900245 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.900459 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.900541 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.903444 (+ 2903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32826: BlockingRecv error: recv got EOF from 127.0.0.1:32826 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 00:19:47 dist-test-slave-777x krb5kdc[7851](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 00:19:47.911991 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:47.904587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32836 (local address 127.0.98.62:43107)
0924 00:19:47.904804 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:47.904806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:47.905009 (+ 203us) server_negotiation.cc:407] Connection header received
0924 00:19:47.905072 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:47.905073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:47.905092 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:47.905154 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:47.905614 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.906083 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.906773 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:47.906958 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:47.907041 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:47.909258 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:47.909259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:47.909259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:47.909267 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:47.910764 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:47.911143 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:47.911144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:47.911145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:47.911228 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:47.911444 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:47.911445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:47.911445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:47.911581 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:47.911698 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:47.911790 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 00:19:47.911877 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
I20260924 00:19:47.913789 7896 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:19:47.913983 7896 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:19:47.915313 8312 tablet_service.cc:1467] Processing CreateTablet for tablet b8494787c93b4859952e2a2962c87367 (DEFAULT_TABLE table=test-table [id=f480a0115f354a0aa71d36b22712a844]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:47.915382 8033 tablet_service.cc:1467] Processing CreateTablet for tablet b8494787c93b4859952e2a2962c87367 (DEFAULT_TABLE table=test-table [id=f480a0115f354a0aa71d36b22712a844]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:47.915453 8312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8494787c93b4859952e2a2962c87367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:47.915501 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8494787c93b4859952e2a2962c87367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:47.916288 8173 tablet_service.cc:1467] Processing CreateTablet for tablet b8494787c93b4859952e2a2962c87367 (DEFAULT_TABLE table=test-table [id=f480a0115f354a0aa71d36b22712a844]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:47.916396 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8494787c93b4859952e2a2962c87367. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:47.916893 8407 tablet_bootstrap.cc:492] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab: Bootstrap starting.
I20260924 00:19:47.917239 8407 tablet_bootstrap.cc:654] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:47.917837 8411 tablet_bootstrap.cc:492] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574: Bootstrap starting.
I20260924 00:19:47.918089 8407 tablet_bootstrap.cc:492] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab: No bootstrap required, opened a new log
I20260924 00:19:47.918136 8407 ts_tablet_manager.cc:1397] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:47.918308 8411 tablet_bootstrap.cc:654] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:47.918280 8407 raft_consensus.cc:348] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.918352 8407 raft_consensus.cc:374] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:47.918376 8407 raft_consensus.cc:729] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac83bdbf50345c2b8df30f54cddd7ab, State: Initialized, Role: FOLLOWER
I20260924 00:19:47.918424 8407 consensus_queue.cc:260] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.918527 8407 ts_tablet_manager.cc:1428] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:47.920269 8408 tablet_bootstrap.cc:492] T b8494787c93b4859952e2a2962c87367 P 2331dfca9ae24769b1aba1eef5e3114a: Bootstrap starting.
I20260924 00:19:47.920706 8408 tablet_bootstrap.cc:654] T b8494787c93b4859952e2a2962c87367 P 2331dfca9ae24769b1aba1eef5e3114a: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:47.920778 8411 tablet_bootstrap.cc:492] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574: No bootstrap required, opened a new log
I20260924 00:19:47.920823 8411 ts_tablet_manager.cc:1397] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 00:19:47.920980 8411 raft_consensus.cc:348] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.921051 8411 raft_consensus.cc:374] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:47.921068 8411 raft_consensus.cc:729] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99153313d5e843eb9d988459ee092574, State: Initialized, Role: FOLLOWER
I20260924 00:19:47.921140 8411 consensus_queue.cc:260] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.921231 8411 ts_tablet_manager.cc:1428] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:47.928815 8408 tablet_bootstrap.cc:492] T b8494787c93b4859952e2a2962c87367 P 2331dfca9ae24769b1aba1eef5e3114a: No bootstrap required, opened a new log
I20260924 00:19:47.928910 8408 ts_tablet_manager.cc:1397] T b8494787c93b4859952e2a2962c87367 P 2331dfca9ae24769b1aba1eef5e3114a: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260924 00:19:47.929075 8408 raft_consensus.cc:348] T b8494787c93b4859952e2a2962c87367 P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.929165 8408 raft_consensus.cc:374] T b8494787c93b4859952e2a2962c87367 P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:47.929194 8408 raft_consensus.cc:729] T b8494787c93b4859952e2a2962c87367 P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2331dfca9ae24769b1aba1eef5e3114a, State: Initialized, Role: FOLLOWER
I20260924 00:19:47.929239 8408 consensus_queue.cc:260] T b8494787c93b4859952e2a2962c87367 P 2331dfca9ae24769b1aba1eef5e3114a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.929340 8408 ts_tablet_manager.cc:1428] T b8494787c93b4859952e2a2962c87367 P 2331dfca9ae24769b1aba1eef5e3114a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 00:19:47.999034 8412 raft_consensus.cc:482] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:19:47.999154 8412 raft_consensus.cc:504] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:47.999320 8412 leader_election.cc:288] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2331dfca9ae24769b1aba1eef5e3114a (127.0.98.1:35223), 99153313d5e843eb9d988459ee092574 (127.0.98.2:41125)
I20260924 00:19:47.999611 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8494787c93b4859952e2a2962c87367" candidate_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99153313d5e843eb9d988459ee092574" is_pre_election: true
I20260924 00:19:47.999644 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8494787c93b4859952e2a2962c87367" candidate_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2331dfca9ae24769b1aba1eef5e3114a" is_pre_election: true
I20260924 00:19:47.999732 8053 raft_consensus.cc:2454] T b8494787c93b4859952e2a2962c87367 P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac83bdbf50345c2b8df30f54cddd7ab in term 0.
I20260924 00:19:47.999732 8193 raft_consensus.cc:2454] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fac83bdbf50345c2b8df30f54cddd7ab in term 0.
I20260924 00:19:47.999938 8265 leader_election.cc:302] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99153313d5e843eb9d988459ee092574, fac83bdbf50345c2b8df30f54cddd7ab; no voters:
I20260924 00:19:48.000046 8412 raft_consensus.cc:2781] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:19:48.000090 8412 raft_consensus.cc:482] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:19:48.000123 8412 raft_consensus.cc:3037] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:48.000886 8412 raft_consensus.cc:504] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:48.001019 8412 leader_election.cc:288] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab [CANDIDATE]: Term 1 election: Requested vote from peers 2331dfca9ae24769b1aba1eef5e3114a (127.0.98.1:35223), 99153313d5e843eb9d988459ee092574 (127.0.98.2:41125)
I20260924 00:19:48.001204 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8494787c93b4859952e2a2962c87367" candidate_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99153313d5e843eb9d988459ee092574"
I20260924 00:19:48.001226 8053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8494787c93b4859952e2a2962c87367" candidate_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2331dfca9ae24769b1aba1eef5e3114a"
I20260924 00:19:48.001304 8193 raft_consensus.cc:3037] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:48.001314 8053 raft_consensus.cc:3037] T b8494787c93b4859952e2a2962c87367 P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:48.001940 8193 raft_consensus.cc:2454] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac83bdbf50345c2b8df30f54cddd7ab in term 1.
I20260924 00:19:48.002120 8265 leader_election.cc:302] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99153313d5e843eb9d988459ee092574, fac83bdbf50345c2b8df30f54cddd7ab; no voters:
I20260924 00:19:48.002200 8053 raft_consensus.cc:2454] T b8494787c93b4859952e2a2962c87367 P 2331dfca9ae24769b1aba1eef5e3114a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fac83bdbf50345c2b8df30f54cddd7ab in term 1.
I20260924 00:19:48.002209 8412 raft_consensus.cc:2781] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:48.002264 8412 raft_consensus.cc:686] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab [term 1 LEADER]: Becoming Leader. State: Replica: fac83bdbf50345c2b8df30f54cddd7ab, State: Running, Role: LEADER
I20260924 00:19:48.002310 8412 consensus_queue.cc:237] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:48.002904 7896 catalog_manager.cc:5495] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab reported cstate change: term changed from 0 to 1, leader changed from <none> to fac83bdbf50345c2b8df30f54cddd7ab (127.0.98.3). New cstate: current_term: 1 leader_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } health_report { overall_health: HEALTHY } } }
I20260924 00:19:48.015544 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:48.013019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.0.98.3:36929)
0924 00:19:48.013067 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:48.013069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:48.013078 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:19:48.013104 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:48.013105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:48.013116 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:48.013165 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:48.013267 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:48.013908 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:48.014790 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:48.015006 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:48.015092 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:48.015254 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:48.015296 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:48.015356 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:48.015426 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 00:19:48.015445 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:19:48.016260 8193 raft_consensus.cc:1261] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574 [term 1 FOLLOWER]: Refusing update from remote peer fac83bdbf50345c2b8df30f54cddd7ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:48.016331 8053 raft_consensus.cc:1261] T b8494787c93b4859952e2a2962c87367 P 2331dfca9ae24769b1aba1eef5e3114a [term 1 FOLLOWER]: Refusing update from remote peer fac83bdbf50345c2b8df30f54cddd7ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:48.016548 8464 consensus_queue.cc:1035] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:48.016650 8412 consensus_queue.cc:1035] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:48.019160 7895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:48.019289 7895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:48.019932 7895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be: Sending DeleteTablet for 3 replicas of tablet b8494787c93b4859952e2a2962c87367
I20260924 00:19:48.020205 8033 tablet_service.cc:1514] Processing DeleteTablet for tablet b8494787c93b4859952e2a2962c87367 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42322
I20260924 00:19:48.020309 8431 tablet_replica.cc:330] T b8494787c93b4859952e2a2962c87367 P 2331dfca9ae24769b1aba1eef5e3114a: stopping tablet replica
I20260924 00:19:48.020365 8431 raft_consensus.cc:2229] T b8494787c93b4859952e2a2962c87367 P 2331dfca9ae24769b1aba1eef5e3114a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:48.020419 8431 raft_consensus.cc:2258] T b8494787c93b4859952e2a2962c87367 P 2331dfca9ae24769b1aba1eef5e3114a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:48.020427 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet b8494787c93b4859952e2a2962c87367 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59752
I20260924 00:19:48.020563 8430 tablet_replica.cc:330] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574: stopping tablet replica
I20260924 00:19:48.020629 8430 raft_consensus.cc:2229] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:48.020668 8430 pending_rounds.cc:62] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574: Trying to abort 2 pending ops.
I20260924 00:19:48.020697 8430 pending_rounds.cc:68] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7332696834112966656 op_type: WRITE_OP write_request { tablet_id: "b8494787c93b4859952e2a2962c87367" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\320\333\321\325\006\"8\n\ntest-admin\022*\n f480a0115f354a0aa71d36b22712a844\020\001\030\001 \001(\001" signature: "<redacted>""*_Rx\202\363\337\360J/?\207\"m\201\336Tx[\n\332\250[\223\324\277`b\210\000\303\367\261l\312\2602D\035\232\313==\2407\201\341\327\356\353\267\357\306\210d\242\241Q{\300q,\017\275" signing_key_seq_num: 0 } } request_id { client_id: "80d4b5b26fe945fd9dfc7038be659cfe" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260924 00:19:48.020896 8430 pending_rounds.cc:68] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332696834057748480 op_type: NO_OP noop_request { }
I20260924 00:19:48.020929 8431 ts_tablet_manager.cc:1905] T b8494787c93b4859952e2a2962c87367 P 2331dfca9ae24769b1aba1eef5e3114a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:48.020946 8430 raft_consensus.cc:2866] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260924 00:19:48.020977 8430 raft_consensus.cc:2258] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:48.021327 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:19:48.021466 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:19:48.022699 8431 ts_tablet_manager.cc:1918] T b8494787c93b4859952e2a2962c87367 P 2331dfca9ae24769b1aba1eef5e3114a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:19:48.022769 8431 log.cc:1196] T b8494787c93b4859952e2a2962c87367 P 2331dfca9ae24769b1aba1eef5e3114a: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-0/wal/wals/b8494787c93b4859952e2a2962c87367
I20260924 00:19:48.023010 8431 ts_tablet_manager.cc:1939] T b8494787c93b4859952e2a2962c87367 P 2331dfca9ae24769b1aba1eef5e3114a: Deleting consensus metadata
I20260924 00:19:48.022992 8173 tablet_service.cc:1467] Processing CreateTablet for tablet 38a0e90bb7c64f5db4f48a813b2f70d3 (DEFAULT_TABLE table=test-table [id=72f8766d74ee453d82273f4f48e200bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:48.023116 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a0e90bb7c64f5db4f48a813b2f70d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:48.023283 8033 tablet_service.cc:1467] Processing CreateTablet for tablet 38a0e90bb7c64f5db4f48a813b2f70d3 (DEFAULT_TABLE table=test-table [id=72f8766d74ee453d82273f4f48e200bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:48.023324 7881 catalog_manager.cc:4841] TS 2331dfca9ae24769b1aba1eef5e3114a (127.0.98.1:35223): tablet b8494787c93b4859952e2a2962c87367 (table test-table [id=f480a0115f354a0aa71d36b22712a844]) successfully deleted
I20260924 00:19:48.023370 8033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a0e90bb7c64f5db4f48a813b2f70d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:48.024503 8408 tablet_bootstrap.cc:492] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a: Bootstrap starting.
I20260924 00:19:48.024586 8411 tablet_bootstrap.cc:492] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 99153313d5e843eb9d988459ee092574: Bootstrap starting.
I20260924 00:19:48.024864 8408 tablet_bootstrap.cc:654] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:48.024912 8411 tablet_bootstrap.cc:654] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 99153313d5e843eb9d988459ee092574: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:48.025393 8408 tablet_bootstrap.cc:492] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a: No bootstrap required, opened a new log
I20260924 00:19:48.025449 8408 ts_tablet_manager.cc:1397] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:48.025552 8408 raft_consensus.cc:348] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:48.025612 8408 raft_consensus.cc:374] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:48.025631 8408 raft_consensus.cc:729] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2331dfca9ae24769b1aba1eef5e3114a, State: Initialized, Role: FOLLOWER
I20260924 00:19:48.025671 8408 consensus_queue.cc:260] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:48.025669 8312 tablet_service.cc:1514] Processing DeleteTablet for tablet b8494787c93b4859952e2a2962c87367 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36780
I20260924 00:19:48.025743 8408 ts_tablet_manager.cc:1428] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:48.025744 8313 tablet_service.cc:1467] Processing CreateTablet for tablet 38a0e90bb7c64f5db4f48a813b2f70d3 (DEFAULT_TABLE table=test-table [id=72f8766d74ee453d82273f4f48e200bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:19:48.025787 8429 tablet_replica.cc:330] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab: stopping tablet replica
I20260924 00:19:48.025831 8429 raft_consensus.cc:2229] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:19:48.025852 8313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38a0e90bb7c64f5db4f48a813b2f70d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:48.025915 8429 raft_consensus.cc:2258] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:48.026302 8411 tablet_bootstrap.cc:492] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 99153313d5e843eb9d988459ee092574: No bootstrap required, opened a new log
I20260924 00:19:48.026356 8411 ts_tablet_manager.cc:1397] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 99153313d5e843eb9d988459ee092574: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 00:19:48.026489 8411 raft_consensus.cc:348] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:48.026563 8411 raft_consensus.cc:374] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:48.026588 8411 raft_consensus.cc:729] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99153313d5e843eb9d988459ee092574, State: Initialized, Role: FOLLOWER
I20260924 00:19:48.026635 8411 consensus_queue.cc:260] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 99153313d5e843eb9d988459ee092574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:48.026743 8411 ts_tablet_manager.cc:1428] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 99153313d5e843eb9d988459ee092574: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 00:19:48.027400 8407 tablet_bootstrap.cc:492] T 38a0e90bb7c64f5db4f48a813b2f70d3 P fac83bdbf50345c2b8df30f54cddd7ab: Bootstrap starting.
I20260924 00:19:48.027724 8407 tablet_bootstrap.cc:654] T 38a0e90bb7c64f5db4f48a813b2f70d3 P fac83bdbf50345c2b8df30f54cddd7ab: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:48.028316 8407 tablet_bootstrap.cc:492] T 38a0e90bb7c64f5db4f48a813b2f70d3 P fac83bdbf50345c2b8df30f54cddd7ab: No bootstrap required, opened a new log
I20260924 00:19:48.028365 8407 ts_tablet_manager.cc:1397] T 38a0e90bb7c64f5db4f48a813b2f70d3 P fac83bdbf50345c2b8df30f54cddd7ab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:48.028515 8407 raft_consensus.cc:348] T 38a0e90bb7c64f5db4f48a813b2f70d3 P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:48.028575 8407 raft_consensus.cc:374] T 38a0e90bb7c64f5db4f48a813b2f70d3 P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:48.028594 8407 raft_consensus.cc:729] T 38a0e90bb7c64f5db4f48a813b2f70d3 P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac83bdbf50345c2b8df30f54cddd7ab, State: Initialized, Role: FOLLOWER
I20260924 00:19:48.028635 8407 consensus_queue.cc:260] T 38a0e90bb7c64f5db4f48a813b2f70d3 P fac83bdbf50345c2b8df30f54cddd7ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:48.028733 8407 ts_tablet_manager.cc:1428] T 38a0e90bb7c64f5db4f48a813b2f70d3 P fac83bdbf50345c2b8df30f54cddd7ab: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:48.029172 8429 ts_tablet_manager.cc:1905] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:48.030582 8430 ts_tablet_manager.cc:1905] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:48.030840 8429 ts_tablet_manager.cc:1918] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:19:48.030892 8429 log.cc:1196] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/b8494787c93b4859952e2a2962c87367
I20260924 00:19:48.031100 8429 ts_tablet_manager.cc:1939] T b8494787c93b4859952e2a2962c87367 P fac83bdbf50345c2b8df30f54cddd7ab: Deleting consensus metadata
I20260924 00:19:48.031477 7882 catalog_manager.cc:4841] TS fac83bdbf50345c2b8df30f54cddd7ab (127.0.98.3:36929): tablet b8494787c93b4859952e2a2962c87367 (table test-table [id=f480a0115f354a0aa71d36b22712a844]) successfully deleted
I20260924 00:19:48.032104 8430 ts_tablet_manager.cc:1918] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:19:48.032177 8430 log.cc:1196] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-1/wal/wals/b8494787c93b4859952e2a2962c87367
I20260924 00:19:48.032387 8430 ts_tablet_manager.cc:1939] T b8494787c93b4859952e2a2962c87367 P 99153313d5e843eb9d988459ee092574: Deleting consensus metadata
I20260924 00:19:48.032794 7881 catalog_manager.cc:4841] TS 99153313d5e843eb9d988459ee092574 (127.0.98.2:41125): tablet b8494787c93b4859952e2a2962c87367 (table test-table [id=f480a0115f354a0aa71d36b22712a844]) successfully deleted
I20260924 00:19:48.048045 8437 raft_consensus.cc:482] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:19:48.048146 8437 raft_consensus.cc:504] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:48.048429 8437 leader_election.cc:288] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99153313d5e843eb9d988459ee092574 (127.0.98.2:41125), fac83bdbf50345c2b8df30f54cddd7ab (127.0.98.3:36929)
I20260924 00:19:48.048630 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a0e90bb7c64f5db4f48a813b2f70d3" candidate_uuid: "2331dfca9ae24769b1aba1eef5e3114a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" is_pre_election: true
I20260924 00:19:48.048678 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a0e90bb7c64f5db4f48a813b2f70d3" candidate_uuid: "2331dfca9ae24769b1aba1eef5e3114a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99153313d5e843eb9d988459ee092574" is_pre_election: true
I20260924 00:19:48.048756 8193 raft_consensus.cc:2454] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2331dfca9ae24769b1aba1eef5e3114a in term 0.
I20260924 00:19:48.048760 8333 raft_consensus.cc:2454] T 38a0e90bb7c64f5db4f48a813b2f70d3 P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2331dfca9ae24769b1aba1eef5e3114a in term 0.
I20260924 00:19:48.048936 7985 leader_election.cc:302] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2331dfca9ae24769b1aba1eef5e3114a, 99153313d5e843eb9d988459ee092574; no voters:
I20260924 00:19:48.049079 8437 raft_consensus.cc:2781] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:19:48.049124 8437 raft_consensus.cc:482] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:19:48.049144 8437 raft_consensus.cc:3037] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:48.049942 8437 raft_consensus.cc:504] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:48.050110 8437 leader_election.cc:288] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a [CANDIDATE]: Term 1 election: Requested vote from peers 99153313d5e843eb9d988459ee092574 (127.0.98.2:41125), fac83bdbf50345c2b8df30f54cddd7ab (127.0.98.3:36929)
I20260924 00:19:48.050276 8193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a0e90bb7c64f5db4f48a813b2f70d3" candidate_uuid: "2331dfca9ae24769b1aba1eef5e3114a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99153313d5e843eb9d988459ee092574"
I20260924 00:19:48.050366 8193 raft_consensus.cc:3037] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 99153313d5e843eb9d988459ee092574 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:48.051062 8193 raft_consensus.cc:2454] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 99153313d5e843eb9d988459ee092574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2331dfca9ae24769b1aba1eef5e3114a in term 1.
I20260924 00:19:48.051201 8333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38a0e90bb7c64f5db4f48a813b2f70d3" candidate_uuid: "2331dfca9ae24769b1aba1eef5e3114a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fac83bdbf50345c2b8df30f54cddd7ab"
I20260924 00:19:48.051296 8333 raft_consensus.cc:3037] T 38a0e90bb7c64f5db4f48a813b2f70d3 P fac83bdbf50345c2b8df30f54cddd7ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:48.051338 7985 leader_election.cc:302] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2331dfca9ae24769b1aba1eef5e3114a, 99153313d5e843eb9d988459ee092574; no voters:
I20260924 00:19:48.051440 8437 raft_consensus.cc:2781] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:48.051538 8437 raft_consensus.cc:686] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a [term 1 LEADER]: Becoming Leader. State: Replica: 2331dfca9ae24769b1aba1eef5e3114a, State: Running, Role: LEADER
I20260924 00:19:48.051608 8437 consensus_queue.cc:237] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } }
I20260924 00:19:48.052095 8333 raft_consensus.cc:2454] T 38a0e90bb7c64f5db4f48a813b2f70d3 P fac83bdbf50345c2b8df30f54cddd7ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2331dfca9ae24769b1aba1eef5e3114a in term 1.
I20260924 00:19:48.052341 7896 catalog_manager.cc:5495] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2331dfca9ae24769b1aba1eef5e3114a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2331dfca9ae24769b1aba1eef5e3114a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2331dfca9ae24769b1aba1eef5e3114a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 } health_report { overall_health: UNKNOWN } } }
I20260924 00:19:48.061813 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:48.059708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42342 (local address 127.0.98.1:35223)
0924 00:19:48.059758 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:48.059759 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:48.059769 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:19:48.059834 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:48.059835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:48.059843 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:48.059886 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:19:48.059965 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:48.060519 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:48.061202 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:48.061384 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:48.061437 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:48.061544 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:19:48.061602 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:19:48.061637 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:48.061692 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 00:19:48.061706 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 00:19:48.064007 8333 raft_consensus.cc:1261] T 38a0e90bb7c64f5db4f48a813b2f70d3 P fac83bdbf50345c2b8df30f54cddd7ab [term 1 FOLLOWER]: Refusing update from remote peer 2331dfca9ae24769b1aba1eef5e3114a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:48.064020 8193 raft_consensus.cc:1261] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 99153313d5e843eb9d988459ee092574 [term 1 FOLLOWER]: Refusing update from remote peer 2331dfca9ae24769b1aba1eef5e3114a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:19:48.064277 8437 consensus_queue.cc:1035] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99153313d5e843eb9d988459ee092574" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:48.064383 8462 consensus_queue.cc:1035] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fac83bdbf50345c2b8df30f54cddd7ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:19:48.074724 7896 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:48.074822 7896 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:19:48.075395 7896 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3244e3d5b0c84d68ae59b5f905e995be: Sending DeleteTablet for 3 replicas of tablet 38a0e90bb7c64f5db4f48a813b2f70d3
I20260924 00:19:48.075670 8313 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a0e90bb7c64f5db4f48a813b2f70d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36780
I20260924 00:19:48.075744 8033 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a0e90bb7c64f5db4f48a813b2f70d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42322
I20260924 00:19:48.075738 8173 tablet_service.cc:1514] Processing DeleteTablet for tablet 38a0e90bb7c64f5db4f48a813b2f70d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:19:48 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59752
I20260924 00:19:48.075775 8429 tablet_replica.cc:330] T 38a0e90bb7c64f5db4f48a813b2f70d3 P fac83bdbf50345c2b8df30f54cddd7ab: stopping tablet replica
I20260924 00:19:48.075825 8429 raft_consensus.cc:2229] T 38a0e90bb7c64f5db4f48a813b2f70d3 P fac83bdbf50345c2b8df30f54cddd7ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:48.075832 8431 tablet_replica.cc:330] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a: stopping tablet replica
I20260924 00:19:48.075870 8429 raft_consensus.cc:2258] T 38a0e90bb7c64f5db4f48a813b2f70d3 P fac83bdbf50345c2b8df30f54cddd7ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:48.075873 8431 raft_consensus.cc:2229] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:19:48.075941 8431 raft_consensus.cc:2258] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:48.076027 8430 tablet_replica.cc:330] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 99153313d5e843eb9d988459ee092574: stopping tablet replica
I20260924 00:19:48.076087 8430 raft_consensus.cc:2229] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 99153313d5e843eb9d988459ee092574 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:19:48.076134 8430 raft_consensus.cc:2258] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 99153313d5e843eb9d988459ee092574 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:19:48.076119 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 7968
I20260924 00:19:48.076259 8431 ts_tablet_manager.cc:1905] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 2331dfca9ae24769b1aba1eef5e3114a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:48.077742 8430 ts_tablet_manager.cc:1905] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 99153313d5e843eb9d988459ee092574: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:48.077740 8429 ts_tablet_manager.cc:1905] T 38a0e90bb7c64f5db4f48a813b2f70d3 P fac83bdbf50345c2b8df30f54cddd7ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:19:48.079238 8430 ts_tablet_manager.cc:1918] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 99153313d5e843eb9d988459ee092574: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:19:48.079320 8430 log.cc:1196] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 99153313d5e843eb9d988459ee092574: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-1/wal/wals/38a0e90bb7c64f5db4f48a813b2f70d3
I20260924 00:19:48.079519 8429 ts_tablet_manager.cc:1918] T 38a0e90bb7c64f5db4f48a813b2f70d3 P fac83bdbf50345c2b8df30f54cddd7ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 00:19:48.079547 8430 ts_tablet_manager.cc:1939] T 38a0e90bb7c64f5db4f48a813b2f70d3 P 99153313d5e843eb9d988459ee092574: Deleting consensus metadata
I20260924 00:19:48.079586 8429 log.cc:1196] T 38a0e90bb7c64f5db4f48a813b2f70d3 P fac83bdbf50345c2b8df30f54cddd7ab: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/38a0e90bb7c64f5db4f48a813b2f70d3
I20260924 00:19:48.079790 8429 ts_tablet_manager.cc:1939] T 38a0e90bb7c64f5db4f48a813b2f70d3 P fac83bdbf50345c2b8df30f54cddd7ab: Deleting consensus metadata
I20260924 00:19:48.079939 7881 catalog_manager.cc:4841] TS 99153313d5e843eb9d988459ee092574 (127.0.98.2:41125): tablet 38a0e90bb7c64f5db4f48a813b2f70d3 (table test-table [id=72f8766d74ee453d82273f4f48e200bf]) successfully deleted
I20260924 00:19:48.080209 7882 catalog_manager.cc:4841] TS fac83bdbf50345c2b8df30f54cddd7ab (127.0.98.3:36929): tablet 38a0e90bb7c64f5db4f48a813b2f70d3 (table test-table [id=72f8766d74ee453d82273f4f48e200bf]) successfully deleted
W20260924 00:19:48.083946 7881 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35223 (error 108)
I20260924 00:19:48.084239 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 8108
I20260924 00:19:48.085822 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:48.085584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35223 (local address 127.0.0.1:42344)
0924 00:19:48.085729 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:48.085785 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
W20260924 00:19:48.086066 7881 catalog_manager.cc:4584] TS 2331dfca9ae24769b1aba1eef5e3114a (127.0.98.1:35223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38a0e90bb7c64f5db4f48a813b2f70d3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35223: connect: Connection refused (error 111)
I20260924 00:19:48.089223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 8248
I20260924 00:19:48.096272 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 7864
2026-09-24T00:19:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2855 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:19:48 dist-test-slave-777x krb5kdc[8496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:19:48 dist-test-slave-777x krb5kdc[8496](info): set up 2 sockets
Sep 24 00:19:48 dist-test-slave-777x krb5kdc[8496](info): commencing operation
krb5kdc: starting...
W20260924 00:19:49.172848 392 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:19:49 dist-test-slave-777x krb5kdc[8496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:19:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:19:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:19:49.303809 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45969
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:49.390908 8509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:49.391173 8509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:49.391216 8509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:49.392584 8509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:19:49.392663 8509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:49.392689 8509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:49.392724 8509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:19:49.392755 8509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:19:49.394441 8509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:49.394722 8509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:49.394955 8509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:49.400344 8514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:49.400309 8517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:49.400302 8515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:49.400650 8509 server_base.cc:1034] running on GCE node
I20260924 00:19:49.400870 8509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:49.401135 8509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:49.402319 8509 hybrid_clock.cc:648] HybridClock initialized: now 1790209189402304 us; error 29 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T00:19:49Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1313 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:19:49 dist-test-slave-777x krb5kdc[8522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:19:49 dist-test-slave-777x krb5kdc[8522](info): set up 2 sockets
Sep 24 00:19:49 dist-test-slave-777x krb5kdc[8522](info): commencing operation
krb5kdc: starting...
W20260924 00:19:50.452899 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:19:50 dist-test-slave-777x krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:19:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:19:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:19:50.575901 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35491
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:50.661211 8535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:50.661444 8535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:50.661481 8535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:50.662736 8535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:19:50.662783 8535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:50.662798 8535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:50.662811 8535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:19:50.662823 8535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:19:50.664430 8535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:50.664698 8535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:50.664959 8535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:50.669965 8540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:50.669986 8543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:50.669997 8541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:50.670156 8535 server_base.cc:1034] running on GCE node
I20260924 00:19:50.670625 8535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:50.670935 8535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:50.672119 8535 hybrid_clock.cc:648] HybridClock initialized: now 1790209190672095 us; error 38 us; skew 500 ppm
Sep 24 00:19:50 dist-test-slave-777x krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:50.682196 8535 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:19:50.682818 8535 webserver.cc:460] Webserver started at http://127.0.98.62:33505/ using document root <none> and password file <none>
I20260924 00:19:50.683064 8535 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:50.683135 8535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:50.683300 8535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:50.684494 8535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "86660b954496457dafc73364e5333e0c"
format_stamp: "Formatted at 2026-09-24 00:19:50 on dist-test-slave-777x"
server_key: "fe95ea16513f5b7d1cefc20a6912c7c6"
server_key_iv: "3570890fc08071f0bf668ad254448da0"
server_key_version: "encryptionkey@0"
I20260924 00:19:50.684877 8535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "86660b954496457dafc73364e5333e0c"
format_stamp: "Formatted at 2026-09-24 00:19:50 on dist-test-slave-777x"
server_key: "fe95ea16513f5b7d1cefc20a6912c7c6"
server_key_iv: "3570890fc08071f0bf668ad254448da0"
server_key_version: "encryptionkey@0"
I20260924 00:19:50.686280 8535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:19:50.687206 8550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:50.687415 8535 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:50.687588 8535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "86660b954496457dafc73364e5333e0c"
format_stamp: "Formatted at 2026-09-24 00:19:50 on dist-test-slave-777x"
server_key: "fe95ea16513f5b7d1cefc20a6912c7c6"
server_key_iv: "3570890fc08071f0bf668ad254448da0"
server_key_version: "encryptionkey@0"
I20260924 00:19:50.687677 8535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:50.705263 8535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:50.707587 8535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:50.707779 8535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:50.712284 8535 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35491
I20260924 00:19:50.712337 8602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35491 every 8 connection(s)
I20260924 00:19:50.712875 8535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:19:50.713580 8603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:19:50.715999 8603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c: Bootstrap starting.
I20260924 00:19:50.716709 8603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c: Neither blocks nor log segments found. Creating new log.
I20260924 00:19:50.716969 8603 log.cc:824] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:50.717788 8603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c: No bootstrap required, opened a new log
I20260924 00:19:50.719496 8603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86660b954496457dafc73364e5333e0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35491 } }
I20260924 00:19:50.719652 8603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:19:50.719678 8603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86660b954496457dafc73364e5333e0c, State: Initialized, Role: FOLLOWER
I20260924 00:19:50.719794 8603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86660b954496457dafc73364e5333e0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35491 } }
I20260924 00:19:50.719888 8603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:19:50.719935 8603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:19:50.719990 8603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:19:50.721200 8603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86660b954496457dafc73364e5333e0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35491 } }
I20260924 00:19:50.721306 8603 leader_election.cc:302] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86660b954496457dafc73364e5333e0c; no voters:
I20260924 00:19:50.721331 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 8535
I20260924 00:19:50.721468 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:19:50.721517 8603 leader_election.cc:288] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:19:50.721662 8606 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:19:50.721696 392 external_mini_cluster.cc:1428] Setting key d4bfc03c7b15715736c5e8204338edec
I20260924 00:19:50.721788 8603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:19:50.721889 8606 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [term 1 LEADER]: Becoming Leader. State: Replica: 86660b954496457dafc73364e5333e0c, State: Running, Role: LEADER
I20260924 00:19:50.722045 8606 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86660b954496457dafc73364e5333e0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35491 } }
I20260924 00:19:50.722492 8608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86660b954496457dafc73364e5333e0c. Latest consensus state: current_term: 1 leader_uuid: "86660b954496457dafc73364e5333e0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86660b954496457dafc73364e5333e0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35491 } } }
I20260924 00:19:50.722503 8607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86660b954496457dafc73364e5333e0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86660b954496457dafc73364e5333e0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35491 } } }
I20260924 00:19:50.722725 8607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:50.722970 8608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [sys.catalog]: This master's current role is: LEADER
W20260924 00:19:50.723815 8623 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:19:50.723876 8623 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:19:50.723977 8617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:19:50.724220 8617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:19:50.726066 8617 catalog_manager.cc:1347] Generated new cluster ID: 3f0a20124e3e4edcab604de11d9837eb
I20260924 00:19:50.726137 8617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:19:50 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:50.735092 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:50.722222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42912 (local address 127.0.98.62:35491)
0924 00:19:50.722493 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:50.722503 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:50.722538 (+ 35us) server_negotiation.cc:407] Connection header received
0924 00:19:50.722668 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:50.722675 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:50.722735 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:50.723683 (+ 948us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:50.723780 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:50.725047 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:50.726443 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:50.727051 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:50.727165 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:50.730371 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:50.730376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:50.730376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:50.730392 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:50.733265 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:50.733705 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:50.733710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:50.733711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:50.733844 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:50.734061 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:50.734062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:50.734063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:50.734427 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:50.734604 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:50.734690 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 00:19:50.734842 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":64,"threads_started":1}
I20260924 00:19:50.754586 8617 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:19:50.755462 8617 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:19:50.760905 8617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c: Generated new TSK 0
I20260924 00:19:50.761122 8617 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:19:50.796821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35491
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:50.888800 8629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:50.889040 8629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:50.889066 8629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:50.890277 8629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:50.890321 8629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:50.890347 8629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:19:50.892256 8629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:50.892480 8629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:50.892675 8629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:19:50.898485 8629 server_base.cc:1034] running on GCE node
W20260924 00:19:50.898463 8634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:50.898463 8635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:50.898466 8637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:50.899109 8629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:50.899396 8629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:50.900591 8629 hybrid_clock.cc:648] HybridClock initialized: now 1790209190900553 us; error 51 us; skew 500 ppm
Sep 24 00:19:50 dist-test-slave-777x krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:50.903723 8629 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:19:50.904381 8629 webserver.cc:460] Webserver started at http://127.0.98.1:43051/ using document root <none> and password file <none>
I20260924 00:19:50.904683 8629 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:50.904763 8629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:50.904929 8629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:50.906191 8629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "eccaf8c05c224ac88b7a788ecb95325e"
format_stamp: "Formatted at 2026-09-24 00:19:50 on dist-test-slave-777x"
server_key: "d661e46b1192a4324675ef6413cf40fb"
server_key_iv: "1f4b69afc2e2dc327bb2f13b69d31cc5"
server_key_version: "encryptionkey@0"
I20260924 00:19:50.906589 8629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "eccaf8c05c224ac88b7a788ecb95325e"
format_stamp: "Formatted at 2026-09-24 00:19:50 on dist-test-slave-777x"
server_key: "d661e46b1192a4324675ef6413cf40fb"
server_key_iv: "1f4b69afc2e2dc327bb2f13b69d31cc5"
server_key_version: "encryptionkey@0"
I20260924 00:19:50.908094 8629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:19:50.909188 8644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:50.909523 8629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:50.909680 8629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "eccaf8c05c224ac88b7a788ecb95325e"
format_stamp: "Formatted at 2026-09-24 00:19:50 on dist-test-slave-777x"
server_key: "d661e46b1192a4324675ef6413cf40fb"
server_key_iv: "1f4b69afc2e2dc327bb2f13b69d31cc5"
server_key_version: "encryptionkey@0"
I20260924 00:19:50.909776 8629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:50.937812 8629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:50.940629 8629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:50.940812 8629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:50.941082 8629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:50.941401 8629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:50.941457 8629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:50.941500 8629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:50.941534 8629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:50.947836 8629 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36699
I20260924 00:19:50.947875 8757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36699 every 8 connection(s)
I20260924 00:19:50.948254 8629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:19:50.952569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 8629
I20260924 00:19:50.952697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:19:50.952898 392 external_mini_cluster.cc:1428] Setting key fc4bce413bb88e186c5fc54e39e56ad1
Sep 24 00:19:50 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:50.959779 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:50.949147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37051 (local address 127.0.98.62:35491)
0924 00:19:50.949212 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:50.949214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:50.949414 (+ 200us) server_negotiation.cc:407] Connection header received
0924 00:19:50.950394 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:50.950396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:50.950406 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:50.950469 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:50.951170 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:50.951700 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:50.953122 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:50.953358 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:50.953442 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:50.956018 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:50.956023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:50.956023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:50.956039 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:50.958022 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:50.958567 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:50.958569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:50.958569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:50.958642 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:50.958974 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:50.958975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:50.958975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:50.959123 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:50.959210 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:50.959569 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 00:19:50.959663 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 00:19:50.960783 8760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:50.949094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.1:37051)
0924 00:19:50.949277 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:50.949321 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:50.950283 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:50.950603 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:50.950607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:50.950881 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:50.951086 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:50.951090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:50.951808 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:50.951809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:50.953016 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:50.953020 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:50.953491 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:50.953492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:50.953615 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:50.954341 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:50.954351 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:50.955887 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:50.958160 (+ 2273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:50.958169 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:50.958172 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:50.958450 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:50.958718 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:50.958719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:50.958720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:50.958845 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:50.959200 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:50.959208 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:50.959416 (+ 208us) client_negotiation.cc:769] Sending connection context
0924 00:19:50.959546 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 00:19:50.960591 (+ 1045us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 00:19:50.961400 8758 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35491
I20260924 00:19:50.961500 8758 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:50.961661 8758 heartbeater.cc:511] Master 127.0.98.62:35491 requested a full tablet report, sending...
I20260924 00:19:50.962344 8567 ts_manager.cc:194] Registered new tserver with Master: eccaf8c05c224ac88b7a788ecb95325e (127.0.98.1:36699)
I20260924 00:19:50.963276 8567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:19:50.980505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35491
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:51.064087 8763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:51.064337 8763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:51.064368 8763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:51.065616 8763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:51.065658 8763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:51.065685 8763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:19:51.067250 8763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:51.067462 8763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:51.067636 8763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:51.072769 8769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:51.072800 8771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:51.072813 8768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:51.072922 8763 server_base.cc:1034] running on GCE node
I20260924 00:19:51.073277 8763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:51.073544 8763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:51.074729 8763 hybrid_clock.cc:648] HybridClock initialized: now 1790209191074719 us; error 32 us; skew 500 ppm
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:51.077670 8763 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:19:51.078188 8763 webserver.cc:460] Webserver started at http://127.0.98.2:41273/ using document root <none> and password file <none>
I20260924 00:19:51.078408 8763 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:51.078454 8763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:51.078550 8763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:51.079661 8763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "b60766d81380430f829b8fbc9d46003e"
format_stamp: "Formatted at 2026-09-24 00:19:51 on dist-test-slave-777x"
server_key: "e93f4a8e354f8c67bce6c9039b6c9e8f"
server_key_iv: "a9948d21e5e0bbecd3d908cb6510f01b"
server_key_version: "encryptionkey@0"
I20260924 00:19:51.080034 8763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b60766d81380430f829b8fbc9d46003e"
format_stamp: "Formatted at 2026-09-24 00:19:51 on dist-test-slave-777x"
server_key: "e93f4a8e354f8c67bce6c9039b6c9e8f"
server_key_iv: "a9948d21e5e0bbecd3d908cb6510f01b"
server_key_version: "encryptionkey@0"
I20260924 00:19:51.081602 8763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:51.082391 8778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:51.082669 8763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:19:51.082746 8763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "b60766d81380430f829b8fbc9d46003e"
format_stamp: "Formatted at 2026-09-24 00:19:51 on dist-test-slave-777x"
server_key: "e93f4a8e354f8c67bce6c9039b6c9e8f"
server_key_iv: "a9948d21e5e0bbecd3d908cb6510f01b"
server_key_version: "encryptionkey@0"
I20260924 00:19:51.082839 8763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:51.093979 8763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:51.096711 8763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:51.096886 8763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:51.097177 8763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:51.097580 8763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:51.097620 8763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:51.097644 8763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:51.097759 8763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:51.103684 8763 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43247
I20260924 00:19:51.103721 8891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43247 every 8 connection(s)
I20260924 00:19:51.104089 8763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:19:51.105767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 8763
I20260924 00:19:51.105886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:19:51.106127 392 external_mini_cluster.cc:1428] Setting key c31560a41f65a64d96cce329b146b4a5
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 00:19:51.116740 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:51.104997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40435 (local address 127.0.98.62:35491)
0924 00:19:51.105068 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:51.105070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:51.105277 (+ 207us) server_negotiation.cc:407] Connection header received
0924 00:19:51.106450 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:51.106452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:51.106464 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:51.106553 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:51.107345 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.108126 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.110008 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.110208 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.110275 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.113434 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:51.113435 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:51.113435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:51.113443 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:51.114997 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:51.115560 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:51.115561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:51.115562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:51.115617 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:51.115941 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:51.115941 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:51.115942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:51.116073 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:51.116222 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:51.116546 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 00:19:51.116644 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 00:19:51.116954 8894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:51.104956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.2:40435)
0924 00:19:51.105120 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:51.105169 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:51.106286 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:51.106630 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:51.106634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:51.107029 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:51.107238 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:51.107244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.108281 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.108283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:51.109876 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:51.109879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.110311 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.110312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:51.110483 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.111202 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:51.111215 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:51.113328 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:51.115100 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:51.115108 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:51.115111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:51.115471 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:51.115692 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:51.115693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:51.115694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:51.115871 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:51.116240 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:51.116246 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:51.116398 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 00:19:51.116529 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 00:19:51.116703 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260924 00:19:51.117631 8892 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35491
I20260924 00:19:51.117745 8892 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:51.117975 8892 heartbeater.cc:511] Master 127.0.98.62:35491 requested a full tablet report, sending...
I20260924 00:19:51.118570 8567 ts_manager.cc:194] Registered new tserver with Master: b60766d81380430f829b8fbc9d46003e (127.0.98.2:43247)
I20260924 00:19:51.119062 8567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40435
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:19:51.140035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35491
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:51.228937 8897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:51.229153 8897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:51.229179 8897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:51.230355 8897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:51.230400 8897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:51.230429 8897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:19:51.231985 8897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:51.232259 8897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:51.232506 8897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:51.237598 8903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:51.237598 8902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:51.237661 8905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:51.237927 8897 server_base.cc:1034] running on GCE node
I20260924 00:19:51.238224 8897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:51.238510 8897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:51.239696 8897 hybrid_clock.cc:648] HybridClock initialized: now 1790209191239678 us; error 32 us; skew 500 ppm
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:51.242473 8897 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:19:51.243011 8897 webserver.cc:460] Webserver started at http://127.0.98.3:36609/ using document root <none> and password file <none>
I20260924 00:19:51.243261 8897 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:51.243333 8897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:51.243490 8897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:19:51.244663 8897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "ced9e6cde47844da8bd0a9a9fb5b4f0d"
format_stamp: "Formatted at 2026-09-24 00:19:51 on dist-test-slave-777x"
server_key: "34f78bc8aa75fec9ba86d410999cb8be"
server_key_iv: "ab51e4eb72a1a14bfbd27e3e09ecf170"
server_key_version: "encryptionkey@0"
I20260924 00:19:51.245026 8897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ced9e6cde47844da8bd0a9a9fb5b4f0d"
format_stamp: "Formatted at 2026-09-24 00:19:51 on dist-test-slave-777x"
server_key: "34f78bc8aa75fec9ba86d410999cb8be"
server_key_iv: "ab51e4eb72a1a14bfbd27e3e09ecf170"
server_key_version: "encryptionkey@0"
I20260924 00:19:51.246419 8897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:19:51.247259 8912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:51.247478 8897 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:51.247581 8897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "ced9e6cde47844da8bd0a9a9fb5b4f0d"
format_stamp: "Formatted at 2026-09-24 00:19:51 on dist-test-slave-777x"
server_key: "34f78bc8aa75fec9ba86d410999cb8be"
server_key_iv: "ab51e4eb72a1a14bfbd27e3e09ecf170"
server_key_version: "encryptionkey@0"
I20260924 00:19:51.247663 8897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:51.275597 8897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:51.278151 8897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:51.278332 8897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:51.278601 8897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:51.278923 8897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:51.278980 8897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:51.279023 8897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:51.279062 8897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:51.285522 8897 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34535
I20260924 00:19:51.285573 9025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34535 every 8 connection(s)
I20260924 00:19:51.285959 8897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:19:51.296191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 8897
I20260924 00:19:51.296310 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:19:51.296499 392 external_mini_cluster.cc:1428] Setting key 1edda1e2805fd4e390acfe3ab3b69294
I20260924 00:19:51.296779 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:51.286810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33485 (local address 127.0.98.62:35491)
0924 00:19:51.286873 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:51.286876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:51.287239 (+ 363us) server_negotiation.cc:407] Connection header received
0924 00:19:51.288385 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:51.288386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:51.288400 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:51.288478 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:51.289178 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.289799 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.290910 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.291132 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.291195 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.293787 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:51.293788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:51.293789 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:51.293795 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:51.295178 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:51.295585 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:51.295586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:51.295586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:51.295643 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:51.295902 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:51.295903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:51.295903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:51.296098 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:51.296199 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:51.296592 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 00:19:51.296674 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:19:51.296989 9028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:51.286806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.3:33485)
0924 00:19:51.287039 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:51.287092 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:51.288157 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:51.288550 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:51.288554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:51.288844 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:51.289067 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:51.289071 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.289955 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.289956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:51.290828 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:51.290831 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.291221 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.291222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:51.291338 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.291989 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:51.291998 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:51.293673 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:19:51.295258 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:51.295264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:51.295266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:51.295496 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:51.295716 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:19:51.295716 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:19:51.295717 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:19:51.295828 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:19:51.296257 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:19:51.296263 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:19:51.296444 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 00:19:51.296544 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 00:19:51.296742 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":113,"threads_started":1}
I20260924 00:19:51.297567 9026 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35491
I20260924 00:19:51.297663 9026 heartbeater.cc:464] Registering TS with master...
I20260924 00:19:51.297801 9026 heartbeater.cc:511] Master 127.0.98.62:35491 requested a full tablet report, sending...
I20260924 00:19:51.298210 8567 ts_manager.cc:194] Registered new tserver with Master: ced9e6cde47844da8bd0a9a9fb5b4f0d (127.0.98.3:34535)
I20260924 00:19:51.298547 8567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33485
I20260924 00:19:51.308447 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 00:19:51.385772 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 8629
I20260924 00:19:51.390867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 8763
I20260924 00:19:51.396010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 8897
I20260924 00:19:51.400940 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 8535
I20260924 00:19:51.406544 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35491
--webserver_interface=127.0.98.62
--webserver_port=33505
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:19:51.487720 9035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:51.487941 9035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:51.487964 9035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:51.489106 9035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:19:51.489145 9035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:51.489159 9035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:51.489172 9035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:19:51.489184 9035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:19:51.490623 9035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:51.490818 9035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:51.490985 9035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:51.495841 9041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:51.495893 9040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:51.495889 9043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:51.496047 9035 server_base.cc:1034] running on GCE node
I20260924 00:19:51.496331 9035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:51.496709 9035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:51.497906 9035 hybrid_clock.cc:648] HybridClock initialized: now 1790209191497883 us; error 44 us; skew 500 ppm
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:51.500698 9035 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:19:51.501187 9035 webserver.cc:460] Webserver started at http://127.0.98.62:33505/ using document root <none> and password file <none>
I20260924 00:19:51.501434 9035 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:51.501523 9035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:51.503013 9035 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:51.503703 9050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:51.503887 9035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:19:51.504020 9035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "86660b954496457dafc73364e5333e0c"
format_stamp: "Formatted at 2026-09-24 00:19:50 on dist-test-slave-777x"
server_key: "fe95ea16513f5b7d1cefc20a6912c7c6"
server_key_iv: "3570890fc08071f0bf668ad254448da0"
server_key_version: "encryptionkey@0"
I20260924 00:19:51.504464 9035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:51.527159 9035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:51.529515 9035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:51.529762 9035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:51.534145 9035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35491
I20260924 00:19:51.534183 9102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35491 every 8 connection(s)
I20260924 00:19:51.534799 9035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:19:51.536507 9103 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:19:51.537200 9103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c: Bootstrap starting.
I20260924 00:19:51.542166 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 9035
I20260924 00:19:51.542570 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36699
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35491
--webserver_port=43051
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:19:51.546571 9103 log.cc:824] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c: Log is configured to *not* fsync() on all Append() calls
I20260924 00:19:51.548518 9103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:19:51.548839 9103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c: Bootstrap complete.
I20260924 00:19:51.551287 9103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86660b954496457dafc73364e5333e0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35491 } }
I20260924 00:19:51.551537 9103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86660b954496457dafc73364e5333e0c, State: Initialized, Role: FOLLOWER
I20260924 00:19:51.551719 9103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86660b954496457dafc73364e5333e0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35491 } }
I20260924 00:19:51.551798 9103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:19:51.551870 9103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:19:51.551923 9103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:19:51.553118 9103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86660b954496457dafc73364e5333e0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35491 } }
I20260924 00:19:51.553249 9103 leader_election.cc:302] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86660b954496457dafc73364e5333e0c; no voters:
I20260924 00:19:51.553474 9103 leader_election.cc:288] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 00:19:51.553515 9107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:19:51.553680 9107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [term 2 LEADER]: Becoming Leader. State: Replica: 86660b954496457dafc73364e5333e0c, State: Running, Role: LEADER
I20260924 00:19:51.553787 9103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:19:51.553876 9107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86660b954496457dafc73364e5333e0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35491 } }
I20260924 00:19:51.554240 9107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86660b954496457dafc73364e5333e0c. Latest consensus state: current_term: 2 leader_uuid: "86660b954496457dafc73364e5333e0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86660b954496457dafc73364e5333e0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35491 } } }
I20260924 00:19:51.554330 9109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "86660b954496457dafc73364e5333e0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86660b954496457dafc73364e5333e0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35491 } } }
I20260924 00:19:51.554582 9109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:51.554553 9107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c [sys.catalog]: This master's current role is: LEADER
I20260924 00:19:51.555096 9116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:19:51.555900 9116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:19:51.556042 9116 catalog_manager.cc:1259] Loaded cluster ID: 3f0a20124e3e4edcab604de11d9837eb
I20260924 00:19:51.556080 9116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:19:51.557368 9116 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:19:51.557564 9116 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 86660b954496457dafc73364e5333e0c: Loaded TSK: 0
I20260924 00:19:51.557785 9116 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 00:19:51.630856 9105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:51.631107 9105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:51.631150 9105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:51.632441 9105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:51.632510 9105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:51.632557 9105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:19:51.634040 9105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43051
--tserver_master_addrs=127.0.98.62:35491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:51.634305 9105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:51.634542 9105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:51.639621 9129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:51.639608 9132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:51.639652 9130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:51.639879 9105 server_base.cc:1034] running on GCE node
I20260924 00:19:51.640110 9105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:51.640407 9105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:51.641600 9105 hybrid_clock.cc:648] HybridClock initialized: now 1790209191641585 us; error 36 us; skew 500 ppm
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209191, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:51.644639 9105 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 00:19:51.645082 9105 webserver.cc:460] Webserver started at http://127.0.98.1:43051/ using document root <none> and password file <none>
I20260924 00:19:51.645316 9105 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:51.645399 9105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:51.646807 9105 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 00:19:51.647536 9139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:51.647691 9105 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:19:51.647770 9105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "eccaf8c05c224ac88b7a788ecb95325e"
format_stamp: "Formatted at 2026-09-24 00:19:50 on dist-test-slave-777x"
server_key: "d661e46b1192a4324675ef6413cf40fb"
server_key_iv: "1f4b69afc2e2dc327bb2f13b69d31cc5"
server_key_version: "encryptionkey@0"
I20260924 00:19:51.648293 9105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:51.666301 9105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:51.669643 9105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:51.669821 9105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:51.670130 9105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:51.670573 9105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:51.670629 9105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:51.670663 9105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:51.670682 9105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:51.678344 9105 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36699
I20260924 00:19:51.678417 9252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36699 every 8 connection(s)
I20260924 00:19:51.678880 9105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:51.688436 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 9105
I20260924 00:19:51.688776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43247
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35491
--webserver_port=41273
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:19:51.689659 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:51.679788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.1:57459)
0924 00:19:51.680047 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:51.680095 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:51.680735 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:51.681225 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:51.681230 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:51.682015 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:51.682289 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:51.682296 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.683576 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.683578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:51.684872 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:51.684876 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.685486 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.685488 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:51.685627 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.686255 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:51.686266 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:51.689381 (+ 3115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":118,"threads_started":1}
W20260924 00:19:51.689810 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:51.679910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57459 (local address 127.0.98.62:35491)
0924 00:19:51.680124 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:51.680142 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:51.680230 (+ 88us) server_negotiation.cc:407] Connection header received
0924 00:19:51.680975 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:51.680982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:51.681032 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:51.681149 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:51.682426 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.683453 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.684972 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.685390 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.685499 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.689584 (+ 4085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57459: BlockingRecv error: recv got EOF from 127.0.98.1:57459 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":95,"threads_started":1}
W20260924 00:19:51.690474 9253 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35491 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35491: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:51.696660 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:51.690747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.1:46593)
0924 00:19:51.690795 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:51.690807 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:51.690851 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:51.691024 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:51.691025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:51.691218 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:51.691282 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:51.691284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.692307 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.692308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:51.693142 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:51.693144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.693621 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.693623 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:51.693720 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.694247 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:51.694255 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:51.696588 (+ 2333us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 00:19:51.697062 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:51.690796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46593 (local address 127.0.98.62:35491)
0924 00:19:51.690858 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:51.690860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:51.690873 (+ 13us) server_negotiation.cc:407] Connection header received
0924 00:19:51.690918 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:51.690919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:51.690932 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:51.690996 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:51.691369 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.692143 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.693245 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.693534 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.693596 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.697007 (+ 3411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46593: BlockingRecv error: recv got EOF from 127.0.98.1:46593 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:51.702571 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:51.697166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.1:37173)
0924 00:19:51.697436 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:51.697451 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:51.697511 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:51.697675 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:51.697676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:51.697848 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:51.697894 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:51.697896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.698716 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.698717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:51.699720 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:51.699723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.700100 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.700101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:51.700201 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.700599 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:51.700607 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:51.702487 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251}
W20260924 00:19:51.702625 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:51.697272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37173 (local address 127.0.98.62:35491)
0924 00:19:51.697301 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:51.697303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:51.697493 (+ 190us) server_negotiation.cc:407] Connection header received
0924 00:19:51.697581 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:51.697581 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:51.697592 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:51.697650 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:51.697983 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.698622 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.699825 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.700013 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.700069 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.702556 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37173: BlockingRecv error: recv got EOF from 127.0.98.1:37173 (error 108)
Metrics: {"server-negotiator.queue_time_us":14,"thread_start_us":124,"threads_started":1}
W20260924 00:19:51.702764 9253 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 00:19:51.775643 9257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:51.775877 9257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:51.775902 9257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:51.777287 9257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:51.777345 9257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:51.777375 9257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:19:51.778923 9257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41273
--tserver_master_addrs=127.0.98.62:35491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:51.779204 9257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:51.779458 9257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:51.784788 9263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:51.784788 9266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:51.784816 9264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:51.784806 9257 server_base.cc:1034] running on GCE node
I20260924 00:19:51.785147 9257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:51.785352 9257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:51.786710 9257 hybrid_clock.cc:648] HybridClock initialized: now 1790209191786692 us; error 29 us; skew 500 ppm
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209191, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:51.789740 9257 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 00:19:51.790166 9257 webserver.cc:460] Webserver started at http://127.0.98.2:41273/ using document root <none> and password file <none>
I20260924 00:19:51.790380 9257 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:51.790447 9257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:51.791872 9257 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:51.792518 9273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:51.792719 9257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:51.792807 9257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "b60766d81380430f829b8fbc9d46003e"
format_stamp: "Formatted at 2026-09-24 00:19:51 on dist-test-slave-777x"
server_key: "e93f4a8e354f8c67bce6c9039b6c9e8f"
server_key_iv: "a9948d21e5e0bbecd3d908cb6510f01b"
server_key_version: "encryptionkey@0"
I20260924 00:19:51.793183 9257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:51.811174 9257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:51.813633 9257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:51.813764 9257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:51.813974 9257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:51.814287 9257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:51.814320 9257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:51.814347 9257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:51.814361 9257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:51.820866 9257 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43247
I20260924 00:19:51.820946 9386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43247 every 8 connection(s)
I20260924 00:19:51.821310 9257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:19:51.824854 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 9257
I20260924 00:19:51.825177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34535
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35491
--webserver_port=36609
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 00:19:51.830627 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:51.822494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47385 (local address 127.0.98.62:35491)
0924 00:19:51.822569 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:51.822571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:51.822760 (+ 189us) server_negotiation.cc:407] Connection header received
0924 00:19:51.823254 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:51.823256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:51.823270 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:51.823343 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:51.824428 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.824941 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.826115 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.826437 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.826517 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.830577 (+ 4060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47385: BlockingRecv error: recv got EOF from 127.0.98.2:47385 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 00:19:51.830717 9389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:51.822342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.2:47385)
0924 00:19:51.822619 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:51.822664 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:51.823157 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:51.823398 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:51.823401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:51.824050 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:51.824337 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:51.824343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.825067 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.825068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:51.826005 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:51.826008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.826623 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.826624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:51.826746 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.827399 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:51.827408 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:51.830435 (+ 3027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":192,"threads_started":1}
W20260924 00:19:51.830978 9387 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35491 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35491: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 00:19:51.835740 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:51.831311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59827 (local address 127.0.98.62:35491)
0924 00:19:51.831372 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:51.831374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:51.831400 (+ 26us) server_negotiation.cc:407] Connection header received
0924 00:19:51.831424 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:51.831425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:51.831436 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:51.831501 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:51.831795 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.832361 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.833058 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.833250 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.833336 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.835697 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59827: BlockingRecv error: recv got EOF from 127.0.98.2:59827 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 00:19:51.835739 9389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:51.831250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.2:59827)
0924 00:19:51.831294 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:51.831304 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:51.831366 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:51.831506 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:51.831506 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:51.831666 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:51.831713 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:51.831714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.832441 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.832442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:51.832968 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:51.832970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.833340 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.833341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:51.833412 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.833910 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:51.833915 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:51.835625 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:51.841058 9389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:51.836211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.2:46091)
0924 00:19:51.836263 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:51.836275 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:51.836320 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:51.836489 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:51.836489 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:51.836602 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:51.836628 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:51.836629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.837675 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.837676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:51.838151 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:51.838152 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.838654 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.838655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:51.838749 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.839249 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:51.839254 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:51.841008 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 00:19:51.841189 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:51.836276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46091 (local address 127.0.98.62:35491)
0924 00:19:51.836361 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:51.836363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:51.836374 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:19:51.836400 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:51.836401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:51.836413 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:51.836487 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:51.836706 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.837590 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.838231 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.838561 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.838632 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.841120 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46091: BlockingRecv error: recv got EOF from 127.0.98.2:46091 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 00:19:51.841522 9387 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 00:19:51.913620 9390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:19:51.913828 9390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:19:51.913852 9390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:19:51.914968 9390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:19:51.915009 9390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:19:51.915035 9390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:19:51.916427 9390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36609
--tserver_master_addrs=127.0.98.62:35491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:19:51.916652 9390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:19:51.916832 9390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:19:51.922066 9398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:51.922098 9396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:19:51.922066 9395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:19:51.922333 9390 server_base.cc:1034] running on GCE node
I20260924 00:19:51.922679 9390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:19:51.922993 9390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:19:51.924237 9390 hybrid_clock.cc:648] HybridClock initialized: now 1790209191924210 us; error 40 us; skew 500 ppm
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209191, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:19:51.926944 9390 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 00:19:51.927353 9390 webserver.cc:460] Webserver started at http://127.0.98.3:36609/ using document root <none> and password file <none>
I20260924 00:19:51.927623 9390 fs_manager.cc:359] Metadata directory not provided
I20260924 00:19:51.927685 9390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:19:51.929234 9390 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:51.929917 9405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:51.930153 9390 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:19:51.930258 9390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "ced9e6cde47844da8bd0a9a9fb5b4f0d"
format_stamp: "Formatted at 2026-09-24 00:19:51 on dist-test-slave-777x"
server_key: "34f78bc8aa75fec9ba86d410999cb8be"
server_key_iv: "ab51e4eb72a1a14bfbd27e3e09ecf170"
server_key_version: "encryptionkey@0"
I20260924 00:19:51.930657 9390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:19:51.948457 9390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:19:51.950552 9390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:19:51.950704 9390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:19:51.950941 9390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:19:51.951285 9390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:19:51.951340 9390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:51.951380 9390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:19:51.951407 9390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:19:51.957746 9390 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34535
I20260924 00:19:51.957791 9518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34535 every 8 connection(s)
I20260924 00:19:51.958144 9390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:19:51.960711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 9390
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 00:19:51.965924 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:51.958880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52845 (local address 127.0.98.62:35491)
0924 00:19:51.958948 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:51.958950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:51.959182 (+ 232us) server_negotiation.cc:407] Connection header received
0924 00:19:51.959729 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:51.959730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:51.959742 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:51.959795 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:51.960865 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.961495 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.962611 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.962803 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.962888 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.965866 (+ 2978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52845: BlockingRecv error: recv got EOF from 127.0.98.3:52845 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 00:19:51.966010 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:51.958892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.3:52845)
0924 00:19:51.959060 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:51.959097 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:51.959618 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:51.959880 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:51.959884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:51.960532 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:51.960730 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:51.960735 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.961663 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.961664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:51.962512 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:51.962514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.962898 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.962900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:51.963027 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.963572 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:51.963582 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:51.965802 (+ 2220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
W20260924 00:19:51.966224 9519 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35491 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35491: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260924 00:19:51.968389 9258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:51.961237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42920 (local address 127.0.98.62:35491)
0924 00:19:51.961324 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:51.961326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:51.961335 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:19:51.961382 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:51.961383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:51.961395 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:51.961472 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:51.962091 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.962570 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.963351 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.963638 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.963713 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.964998 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:19:51.965003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:19:51.965003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:19:51.965015 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:19:51.966957 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:51.967309 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:51.967313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:51.967313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:51.967422 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:19:51.967634 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:19:51.967635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:19:51.967636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:19:51.967865 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:19:51.968012 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 00:19:51.968133 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 00:19:51.968277 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:51.971004 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:51.966452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.3:46581)
0924 00:19:51.966483 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:51.966492 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:51.966558 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:51.966683 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:51.966683 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:51.966807 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:51.966849 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:51.966850 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.967470 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.967471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:51.967991 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:51.967993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.968412 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.968412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:51.968478 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.968798 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:51.968804 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:51.970925 (+ 2121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 00:19:51.971050 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:51.966503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46581 (local address 127.0.98.62:35491)
0924 00:19:51.966559 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:51.966561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:51.966570 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:19:51.966613 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:51.966614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:51.966624 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:51.966677 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:51.966929 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.967381 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.968087 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.968330 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.968411 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.970991 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46581: BlockingRecv error: recv got EOF from 127.0.98.3:46581 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:51 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:51.975618 9521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:51.971315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.3:46953)
0924 00:19:51.971356 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:51.971365 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:51.971402 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:51.971511 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:51.971512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:51.971632 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:51.971657 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:51.971658 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.972295 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.972296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:51.972778 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:51.972779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.973156 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.973157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:51.973209 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.973649 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:51.973654 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:51.975538 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 00:19:51.975666 9256 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:51.971375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46953 (local address 127.0.98.62:35491)
0924 00:19:51.971415 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:51.971416 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:51.971424 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:19:51.971446 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:51.971447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:51.971456 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:51.971499 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:51.971730 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.972222 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.972872 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:51.973071 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:51.973155 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:51.975608 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46953: BlockingRecv error: recv got EOF from 127.0.98.3:46953 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 00:19:51.975742 9519 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 00:19:52 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:52 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:52.709112 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:52.703239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.1:42675)
0924 00:19:52.703427 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:52.703500 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:52.703573 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:52.703735 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:52.703736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:52.703902 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:52.703954 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:52.703955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:52.705109 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:52.705110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:52.705754 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:52.705757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:52.706193 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:52.706194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:52.706242 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:52.706920 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:52.706926 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:52.709028 (+ 2102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":103,"threads_started":1}
W20260924 00:19:52.709319 9524 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:52.703366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42675 (local address 127.0.98.62:35491)
0924 00:19:52.703496 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:52.703499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:52.703572 (+ 73us) server_negotiation.cc:407] Connection header received
0924 00:19:52.703639 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:52.703640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:52.703652 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:52.703716 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:52.704074 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:52.705011 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:52.705853 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:52.706106 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:52.706173 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:52.709225 (+ 3052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42675: BlockingRecv error: recv got EOF from 127.0.98.1:42675 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
Sep 24 00:19:52 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:52 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:52.846871 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:52.841945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.2:41501)
0924 00:19:52.842151 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:52.842165 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:52.842255 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:52.842411 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:52.842412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:52.842609 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:52.842661 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:52.842663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:52.843357 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:52.843358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:52.843909 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:52.843911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:52.844367 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:52.844368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:52.844433 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:52.844961 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:52.844966 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:52.846753 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":123,"threads_started":1}
W20260924 00:19:52.846930 9524 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:52.842054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41501 (local address 127.0.98.62:35491)
0924 00:19:52.842134 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:52.842136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:52.842237 (+ 101us) server_negotiation.cc:407] Connection header received
0924 00:19:52.842329 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:52.842330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:52.842344 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:52.842392 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:52.842751 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:52.843266 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:52.844008 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:52.844280 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:52.844366 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:52.846864 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41501: BlockingRecv error: recv got EOF from 127.0.98.2:41501 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 00:19:52 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:52 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:52.981472 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:52.976251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.3:43835)
0924 00:19:52.976495 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:52.976509 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:52.976610 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:52.976776 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:52.976777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:52.976969 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:52.977022 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:52.977024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:52.977810 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:52.977811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:52.978374 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:52.978376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:52.978770 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:52.978771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:52.978827 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:52.979318 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:52.979323 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:52.981355 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":122,"threads_started":1}
W20260924 00:19:52.981526 9524 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:52.976369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43835 (local address 127.0.98.62:35491)
0924 00:19:52.976478 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:52.976481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:52.976590 (+ 109us) server_negotiation.cc:407] Connection header received
0924 00:19:52.976708 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:52.976709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:52.976723 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:52.976771 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:52.977182 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:52.977713 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:52.978468 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:52.978678 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:52.978768 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:52.981453 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43835: BlockingRecv error: recv got EOF from 127.0.98.3:43835 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 00:19:53 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:53 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:53.715152 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:53.709751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.1:55339)
0924 00:19:53.709897 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:53.709909 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:53.709962 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:53.710179 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:53.710180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:53.710369 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:53.710421 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:53.710423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:53.711262 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:53.711264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:53.711871 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:53.711874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:53.712348 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:53.712349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:53.712406 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:53.712940 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:53.712945 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:53.715089 (+ 2144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
W20260924 00:19:53.715317 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:53.709836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55339 (local address 127.0.98.62:35491)
0924 00:19:53.709981 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:53.709983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:53.709995 (+ 12us) server_negotiation.cc:407] Connection header received
0924 00:19:53.710022 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:53.710022 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:53.710033 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:53.710116 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:53.710610 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:53.711165 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:53.711978 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:53.712269 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:53.712334 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:53.715219 (+ 2885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55339: BlockingRecv error: recv got EOF from 127.0.98.1:55339 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
Sep 24 00:19:53 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:53 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:53.853240 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:53.847487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.2:51197)
0924 00:19:53.847746 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:53.847759 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:53.847858 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:53.848034 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:53.848035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:53.848306 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:53.848357 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:53.848359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:53.849214 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:53.849215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:53.849866 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:53.849869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:53.850261 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:53.850262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:53.850311 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:53.850852 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:53.850856 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:53.853125 (+ 2269us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":155,"threads_started":1}
W20260924 00:19:53.853317 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:53.847668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51197 (local address 127.0.98.62:35491)
0924 00:19:53.847747 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:53.847749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:53.847836 (+ 87us) server_negotiation.cc:407] Connection header received
0924 00:19:53.847928 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:53.847929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:53.847944 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:53.848012 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:53.848446 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:53.849121 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:53.849965 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:53.850159 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:53.850229 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:53.853245 (+ 3016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51197: BlockingRecv error: recv got EOF from 127.0.98.2:51197 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 00:19:53 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:53 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:53.987234 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:53.982147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.3:58383)
0924 00:19:53.982397 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:53.982410 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:53.982481 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:53.982731 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:53.982732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:53.982909 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:53.982955 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:53.982956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:53.983744 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:53.983746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:53.984364 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:53.984367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:53.984736 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:53.984737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:53.984794 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:53.985272 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:53.985277 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:53.987174 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":76,"threads_started":1}
W20260924 00:19:53.987397 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:53.982282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58383 (local address 127.0.98.62:35491)
0924 00:19:53.982370 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:53.982372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:53.982480 (+ 108us) server_negotiation.cc:407] Connection header received
0924 00:19:53.982637 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:53.982638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:53.982652 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:53.982725 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:53.983039 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:53.983644 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:53.984449 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:53.984647 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:53.984734 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:53.987310 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58383: BlockingRecv error: recv got EOF from 127.0.98.3:58383 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 00:19:54 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:54 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:54.720553 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:54.715664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.1:33161)
0924 00:19:54.715826 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:54.715839 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:54.715938 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:54.716138 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:54.716139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:54.716341 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:54.716388 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:54.716390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:54.717138 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:54.717139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:54.717718 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:54.717720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:54.718106 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:54.718107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:54.718171 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:54.718629 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:54.718634 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:54.720436 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":68,"threads_started":1}
W20260924 00:19:54.720597 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:54.715673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33161 (local address 127.0.98.62:35491)
0924 00:19:54.715836 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:54.715839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:54.715917 (+ 78us) server_negotiation.cc:407] Connection header received
0924 00:19:54.716033 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:54.716034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:54.716047 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:54.716132 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:54.716479 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:54.717028 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:54.717820 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:54.718031 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:54.718107 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:54.720523 (+ 2416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33161: BlockingRecv error: recv got EOF from 127.0.98.1:33161 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":97,"threads_started":1}
Sep 24 00:19:54 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:54 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:54.859634 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:54.854088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.2:42333)
0924 00:19:54.854415 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:54.854428 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:54.854539 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:54.854760 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:54.854761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:54.854954 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:54.855031 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:54.855033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:54.855830 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:54.855831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:54.856450 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:54.856453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:54.856874 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:54.856876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:54.856934 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:54.857468 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:54.857474 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:54.859556 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":200,"threads_started":1}
W20260924 00:19:54.859856 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:54.854282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42333 (local address 127.0.98.62:35491)
0924 00:19:54.854371 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:54.854374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:54.854530 (+ 156us) server_negotiation.cc:407] Connection header received
0924 00:19:54.854650 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:54.854651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:54.854666 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:54.854751 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:54.855133 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:54.855718 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:54.856564 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:54.856781 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:54.856873 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:54.859775 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42333: BlockingRecv error: recv got EOF from 127.0.98.2:42333 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 00:19:54 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:54 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:54.997195 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:54.987804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.3:45179)
0924 00:19:54.987955 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:54.987968 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:54.988044 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:54.988259 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:54.988260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:54.988429 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:54.988478 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:54.988479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:54.989235 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:54.989236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:54.989779 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:54.989781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:54.990337 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:54.990338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:54.990402 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:54.995044 (+ 4642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:54.995049 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:54.997125 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
W20260924 00:19:54.997390 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:54.987978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45179 (local address 127.0.98.62:35491)
0924 00:19:54.988045 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:54.988047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:54.988062 (+ 15us) server_negotiation.cc:407] Connection header received
0924 00:19:54.988093 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:54.988094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:54.988111 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:54.988217 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:54.988613 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:54.989152 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:54.989921 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:54.990235 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:54.990314 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:54.997324 (+ 7010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45179: BlockingRecv error: recv got EOF from 127.0.98.3:45179 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 00:19:55 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:55 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:55.730860 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:55.721296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.1:60763)
0924 00:19:55.721456 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:55.721468 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:55.721535 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:55.721707 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:55.721708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:55.721940 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:55.722022 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:55.722025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:55.722981 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:55.722982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:55.723642 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:55.723645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:55.724070 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:55.724071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:55.724121 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:55.728841 (+ 4720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:55.728848 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:55.730797 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
W20260924 00:19:55.731030 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:55.721428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60763 (local address 127.0.98.62:35491)
0924 00:19:55.721555 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:55.721557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:55.721574 (+ 17us) server_negotiation.cc:407] Connection header received
0924 00:19:55.721598 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:55.721599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:55.721610 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:55.721672 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:55.722145 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:55.722886 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:55.723757 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:55.723955 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:55.724021 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:55.730919 (+ 6898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60763: BlockingRecv error: recv got EOF from 127.0.98.1:60763 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
Sep 24 00:19:55 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:55 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:55.865125 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:55.860149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.2:51623)
0924 00:19:55.860306 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:55.860318 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:55.860377 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:55.860513 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:55.860514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:55.860689 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:55.860750 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:55.860752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:55.861391 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:55.861392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:55.861907 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:55.861908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:55.862362 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:55.862363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:55.862415 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:55.863043 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:55.863049 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:55.865025 (+ 1976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":104,"threads_started":1}
W20260924 00:19:55.865213 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:55.860296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51623 (local address 127.0.98.62:35491)
0924 00:19:55.860360 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:55.860362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:55.860371 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:19:55.860428 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:55.860429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:55.860440 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:55.860488 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:55.860820 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:55.861324 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:55.862033 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:55.862247 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:55.862311 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:55.865138 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51623: BlockingRecv error: recv got EOF from 127.0.98.2:51623 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 00:19:55 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:55 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:56.003085 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:55.997765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.3:43399)
0924 00:19:55.997944 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:55.997962 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:55.998041 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:55.998382 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:55.998383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:55.998641 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:55.998697 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:55.998700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:55.999499 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:55.999501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:56.000058 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:56.000061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:56.000470 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:56.000471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:56.000522 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:56.001036 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:56.001041 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:56.003024 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":63,"threads_started":1}
W20260924 00:19:56.003261 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:55.997840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43399 (local address 127.0.98.62:35491)
0924 00:19:55.997912 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:55.997916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:55.998046 (+ 130us) server_negotiation.cc:407] Connection header received
0924 00:19:55.998138 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:55.998139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:55.998169 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:55.998240 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:55.998846 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:55.999392 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:56.000190 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:56.000394 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:56.000457 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:56.003165 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43399: BlockingRecv error: recv got EOF from 127.0.98.3:43399 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 00:19:56 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:56 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:56.737430 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:56.731475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.1:56087)
0924 00:19:56.731703 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:56.731723 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:56.731831 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:56.732045 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:56.732046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:56.732322 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:56.732372 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:56.732375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:56.733302 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:56.733303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:56.734076 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:56.734079 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:56.734612 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:56.734613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:56.734662 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:56.735147 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:56.735169 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:56.737368 (+ 2199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":159,"threads_started":1}
W20260924 00:19:56.737588 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:56.731587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56087 (local address 127.0.98.62:35491)
0924 00:19:56.731732 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:56.731734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:56.731810 (+ 76us) server_negotiation.cc:407] Connection header received
0924 00:19:56.731905 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:56.731905 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:56.731924 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:56.732005 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:56.732455 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:56.733218 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:56.734203 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:56.734533 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:56.734599 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:56.737492 (+ 2893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56087: BlockingRecv error: recv got EOF from 127.0.98.1:56087 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
Sep 24 00:19:56 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:56 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:56.870568 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:56.865763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.2:42743)
0924 00:19:56.865910 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:56.865922 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:56.865982 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:56.866144 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:56.866145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:56.866325 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:56.866372 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:56.866373 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:56.867089 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:56.867090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:56.867647 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:56.867649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:56.868034 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:56.868034 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:56.868076 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:56.868647 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:56.868651 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:56.870455 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":69,"threads_started":1}
W20260924 00:19:56.870607 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:56.865915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42743 (local address 127.0.98.62:35491)
0924 00:19:56.865963 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:56.865965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:56.865997 (+ 32us) server_negotiation.cc:407] Connection header received
0924 00:19:56.866049 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:56.866050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:56.866062 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:56.866127 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:56.866466 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:56.867001 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:56.867748 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:56.867957 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:56.868027 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:56.870542 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42743: BlockingRecv error: recv got EOF from 127.0.98.2:42743 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 24 00:19:57 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:57 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:57.008961 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:57.003678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.3:34509)
0924 00:19:57.003858 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:57.003871 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:57.003942 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:57.004085 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:57.004086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:57.004280 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:57.004327 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:57.004329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:57.005000 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:57.005001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:57.005553 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:57.005555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:57.005964 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:57.005965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:57.006008 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:57.006447 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:57.006453 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:57.008839 (+ 2386us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":132,"threads_started":1}
W20260924 00:19:57.009009 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:57.003780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34509 (local address 127.0.98.62:35491)
0924 00:19:57.003862 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:57.003865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:57.003944 (+ 79us) server_negotiation.cc:407] Connection header received
0924 00:19:57.004002 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:57.004003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:57.004017 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:57.004064 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:57.004396 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:57.004929 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:57.005694 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:57.005885 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:57.005946 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:57.008935 (+ 2989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34509: BlockingRecv error: recv got EOF from 127.0.98.3:34509 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 00:19:57 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:57 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:57.743052 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:57.737995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.1:41097)
0924 00:19:57.738133 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:57.738144 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:57.738199 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:57.738317 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:57.738318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:57.738476 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:57.738526 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:57.738528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:57.739313 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:57.739314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:57.739849 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:57.739851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:57.740315 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:57.740317 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:57.740373 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:57.740966 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:57.740972 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:57.742980 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
W20260924 00:19:57.743198 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:57.738072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41097 (local address 127.0.98.62:35491)
0924 00:19:57.738175 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:57.738177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:57.738188 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:19:57.738245 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:57.738245 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:57.738255 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:57.738312 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:57.738666 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:57.739226 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:57.739939 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:57.740160 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:57.740307 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:57.743082 (+ 2775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41097: BlockingRecv error: recv got EOF from 127.0.98.1:41097 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":49,"threads_started":1}
Sep 24 00:19:57 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:57 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:57.876425 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:57.871248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.2:41835)
0924 00:19:57.871382 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:57.871393 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:57.871492 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:57.871651 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:57.871651 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:57.871848 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:57.871896 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:57.871897 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:57.872750 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:57.872751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:57.873324 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:57.873325 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:57.873701 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:57.873702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:57.873770 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:57.874299 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:57.874304 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:57.876322 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
W20260924 00:19:57.876482 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:57.871301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41835 (local address 127.0.98.62:35491)
0924 00:19:57.871383 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:57.871386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:57.871478 (+ 92us) server_negotiation.cc:407] Connection header received
0924 00:19:57.871556 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:57.871557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:57.871570 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:57.871645 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:57.872025 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:57.872652 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:57.873417 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:57.873611 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:57.873690 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:57.876417 (+ 2727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41835: BlockingRecv error: recv got EOF from 127.0.98.2:41835 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 00:19:58 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:58 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:58.014716 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:58.009544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.3:59831)
0924 00:19:58.009769 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:58.009782 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:58.009848 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:58.010026 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:58.010027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:58.010195 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:58.010245 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:58.010246 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:58.010943 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:58.010944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:58.011634 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:58.011636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:58.012054 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:58.012055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:58.012103 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:58.012729 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:58.012734 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:58.014613 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":142,"threads_started":1}
W20260924 00:19:58.014801 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:58.009734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59831 (local address 127.0.98.62:35491)
0924 00:19:58.009816 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:58.009818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:58.009835 (+ 17us) server_negotiation.cc:407] Connection header received
0924 00:19:58.009911 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:58.009912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:58.009924 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:58.009981 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:58.010318 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:58.010854 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:58.011735 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:58.011953 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:58.012019 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:58.014701 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59831: BlockingRecv error: recv got EOF from 127.0.98.3:59831 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 00:19:58 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:58 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:58.748438 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:58.743583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.1:54247)
0924 00:19:58.743775 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:58.743788 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:58.743851 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:58.744024 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:58.744025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:58.744221 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:58.744271 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:58.744273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:58.744945 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:58.744945 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:58.745536 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:58.745537 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:58.745934 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:58.745935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:58.745980 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:58.746432 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:58.746437 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:58.748375 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
W20260924 00:19:58.748577 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:58.743716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54247 (local address 127.0.98.62:35491)
0924 00:19:58.743874 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:58.743876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:58.743885 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:19:58.743909 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:58.743910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:58.743922 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:58.743993 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:58.744342 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:58.744867 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:58.745624 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:58.745858 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:58.745917 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:58.748483 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54247: BlockingRecv error: recv got EOF from 127.0.98.1:54247 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":54,"threads_started":1}
Sep 24 00:19:58 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:58 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:58.882436 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:58.877046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.2:54951)
0924 00:19:58.877261 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:58.877276 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:58.877365 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:58.877532 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:58.877533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:58.877739 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:58.877790 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:58.877793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:58.878482 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:58.878483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:58.879019 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:58.879021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:58.879403 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:58.879404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:58.879463 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:58.879941 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:58.879946 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:58.882339 (+ 2393us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":104,"threads_started":1}
W20260924 00:19:58.882481 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:58.877180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54951 (local address 127.0.98.62:35491)
0924 00:19:58.877270 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:58.877273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:58.877344 (+ 71us) server_negotiation.cc:407] Connection header received
0924 00:19:58.877428 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:58.877428 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:58.877441 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:58.877525 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:58.877881 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:58.878394 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:58.879113 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:58.879314 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:58.879403 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:58.882418 (+ 3015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54951: BlockingRecv error: recv got EOF from 127.0.98.2:54951 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 00:19:59 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:59 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:59.020553 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:59.015369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.3:37975)
0924 00:19:59.015534 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:59.015554 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:59.015655 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:59.015846 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:59.015847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:59.016138 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:59.016238 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:59.016241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:59.016945 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:59.016946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:59.017582 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:59.017584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:59.017969 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:59.017970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:59.018023 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:59.018573 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:59.018578 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:59.020493 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":98,"threads_started":1}
W20260924 00:19:59.020676 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:59.015487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37975 (local address 127.0.98.62:35491)
0924 00:19:59.015542 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:59.015545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:59.015684 (+ 139us) server_negotiation.cc:407] Connection header received
0924 00:19:59.015722 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:59.015723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:59.015738 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:59.015817 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:59.016322 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:59.016860 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:59.017680 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:59.017872 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:59.017933 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:59.020591 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37975: BlockingRecv error: recv got EOF from 127.0.98.3:37975 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 24 00:19:59 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:59 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:59.754542 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:59.749125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.1:42221)
0924 00:19:59.749278 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:59.749331 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:59.749386 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:59.749596 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:59.749596 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:59.749758 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:59.749808 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:59.749810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:59.750519 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:59.750520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:59.751065 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:59.751067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:59.751490 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:59.751491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:59.751534 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:59.752112 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:59.752117 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:59.754483 (+ 2366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
W20260924 00:19:59.754765 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:59.749278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42221 (local address 127.0.98.62:35491)
0924 00:19:59.749447 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:59.749449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:59.749459 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:19:59.749486 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:59.749487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:59.749498 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:59.749575 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:59.749913 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:59.750434 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:59.751181 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:59.751403 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:59.751475 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:59.754601 (+ 3126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42221: BlockingRecv error: recv got EOF from 127.0.98.1:42221 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
Sep 24 00:19:59 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:19:59 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:19:59.888469 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:19:59.883038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.2:48113)
0924 00:19:59.883275 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 00:19:59.883289 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:19:59.883416 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:19:59.883575 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:19:59.883576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:19:59.883776 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:19:59.883840 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:59.883842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:59.884738 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:59.884739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:59.885262 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:19:59.885263 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:19:59.885728 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:19:59.885729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:19:59.885797 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:59.886365 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:19:59.886371 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:19:59.888406 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":126,"threads_started":1}
W20260924 00:19:59.888620 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:19:59.883148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48113 (local address 127.0.98.62:35491)
0924 00:19:59.883263 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 00:19:59.883266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:19:59.883399 (+ 133us) server_negotiation.cc:407] Connection header received
0924 00:19:59.883479 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:19:59.883480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:19:59.883495 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:19:59.883554 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:19:59.883932 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:59.884644 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:59.885373 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:19:59.885637 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:19:59.885726 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:19:59.888532 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48113: BlockingRecv error: recv got EOF from 127.0.98.2:48113 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 24 00:20:00 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:20:00 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:20:00.026523 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:00.021103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.3:36917)
0924 00:20:00.021260 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:00.021303 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:00.021384 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:00.021561 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:00.021562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:00.021759 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:00.021809 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:00.021810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:00.022550 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:00.022551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:00.023157 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:00.023159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:00.023568 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:00.023569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:00.023631 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:00.024220 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:00.024226 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:00.026421 (+ 2195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":106,"threads_started":1}
W20260924 00:20:00.026582 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:00.021251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36917 (local address 127.0.98.62:35491)
0924 00:20:00.021350 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:00.021353 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:00.021363 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:20:00.021458 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:00.021459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:00.021473 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:00.021555 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:00.021900 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:00.022457 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:00.023275 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:00.023488 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:00.023571 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:00.026506 (+ 2935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36917: BlockingRecv error: recv got EOF from 127.0.98.3:36917 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 00:20:00 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:20:00 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:20:00.764437 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:00.754967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.1:59905)
0924 00:20:00.755104 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:00.755115 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:00.755190 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:00.755318 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:00.755319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:00.755511 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:00.755561 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:00.755563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:00.756292 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:00.756293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:00.756897 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:00.756899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:00.757270 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:00.757271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:00.757314 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:00.762154 (+ 4840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:00.762167 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:00.764370 (+ 2203us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
W20260924 00:20:00.764613 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:00.755047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59905 (local address 127.0.98.62:35491)
0924 00:20:00.755150 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:00.755152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:00.755178 (+ 26us) server_negotiation.cc:407] Connection header received
0924 00:20:00.755241 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:00.755245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:00.755256 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:00.755313 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:00.755645 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:00.756216 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:00.756976 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:00.757167 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:00.757221 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:00.764521 (+ 7300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59905: BlockingRecv error: recv got EOF from 127.0.98.1:59905 (error 108)
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":44,"threads_started":1}
Sep 24 00:20:00 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:20:00 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:20:00.894464 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:00.889066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.2:53411)
0924 00:20:00.889315 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:00.889327 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:00.889433 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:00.889578 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:00.889579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:00.889782 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:00.889833 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:00.889834 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:00.890616 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:00.890617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:00.891228 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:00.891230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:00.891678 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:00.891679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:00.891747 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:00.892360 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:00.892365 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:00.894353 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":118,"threads_started":1}
W20260924 00:20:00.894532 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:00.889189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53411 (local address 127.0.98.62:35491)
0924 00:20:00.889287 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:00.889290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:00.889423 (+ 133us) server_negotiation.cc:407] Connection header received
0924 00:20:00.889508 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:00.889509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:00.889522 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:00.889574 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:00.889922 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:00.890513 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:00.891355 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:00.891579 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:00.891677 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:00.894451 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53411: BlockingRecv error: recv got EOF from 127.0.98.2:53411 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 00:20:01 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:20:01 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:20:01.032758 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:01.027120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.3:44245)
0924 00:20:01.027303 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:01.027316 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:01.027392 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:01.027563 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:01.027564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:01.027749 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:01.027863 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:01.027866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:01.028640 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:01.028642 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:01.029176 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:01.029177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:01.029563 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:01.029563 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:01.029624 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:01.030214 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:01.030220 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:01.032633 (+ 2413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":96,"threads_started":1}
W20260924 00:20:01.032796 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:01.027241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44245 (local address 127.0.98.62:35491)
0924 00:20:01.027353 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:01.027356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:01.027371 (+ 15us) server_negotiation.cc:407] Connection header received
0924 00:20:01.027480 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:01.027481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:01.027497 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:01.027545 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:01.027956 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:01.028550 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:01.029269 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:01.029470 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:01.029562 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:01.032717 (+ 3155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44245: BlockingRecv error: recv got EOF from 127.0.98.3:44245 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 00:20:01 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:20:01 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:20:01.770040 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:01.765041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.1:49353)
0924 00:20:01.765206 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:01.765219 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:01.765276 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:01.765550 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:01.765551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:01.765791 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:01.765839 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:01.765841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:01.766541 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:01.766542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:01.767156 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:01.767158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:01.767530 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:01.767531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:01.767590 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:01.768258 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:01.768266 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:01.769980 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":101,"threads_started":1}
W20260924 00:20:01.770175 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:01.765222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49353 (local address 127.0.98.62:35491)
0924 00:20:01.765370 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:01.765372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:01.765381 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:20:01.765407 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:01.765408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:01.765418 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:01.765493 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:01.765948 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:01.766449 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:01.767245 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:01.767450 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:01.767506 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:01.770083 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49353: BlockingRecv error: recv got EOF from 127.0.98.1:49353 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
Sep 24 00:20:01 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:20:01 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:20:01.900836 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:01.895011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.2:33757)
0924 00:20:01.895196 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:01.895208 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:01.895300 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:01.895505 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:01.895506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:01.895714 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:01.895769 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:01.895772 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:01.896662 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:01.896663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:01.897255 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:01.897257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:01.897684 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:01.897685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:01.897749 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:01.898325 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:01.898330 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:01.900723 (+ 2393us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":92,"threads_started":1}
W20260924 00:20:01.901077 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:01.895102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33757 (local address 127.0.98.62:35491)
0924 00:20:01.895174 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:01.895176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:01.895279 (+ 103us) server_negotiation.cc:407] Connection header received
0924 00:20:01.895410 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:01.895411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:01.895422 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:01.895497 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:01.895942 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:01.896571 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:01.897354 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:01.897586 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:01.897682 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:01.901013 (+ 3331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33757: BlockingRecv error: recv got EOF from 127.0.98.2:33757 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 00:20:02 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:20:02 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:20:02.039435 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:02.033492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.3:47569)
0924 00:20:02.033688 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:02.033701 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:02.033805 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:02.034003 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:02.034005 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:02.034260 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:02.034310 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:02.034312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:02.035112 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:02.035113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:02.035711 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:02.035713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:02.036125 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:02.036126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:02.036225 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:02.036798 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:02.036802 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:02.039333 (+ 2531us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":80,"threads_started":1}
W20260924 00:20:02.039508 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:02.033593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47569 (local address 127.0.98.62:35491)
0924 00:20:02.033672 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:02.033675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:02.033781 (+ 106us) server_negotiation.cc:407] Connection header received
0924 00:20:02.033887 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:02.033888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:02.033911 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:02.033980 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:02.034432 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:02.034997 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:02.035826 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:02.036039 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:02.036127 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:02.039438 (+ 3311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47569: BlockingRecv error: recv got EOF from 127.0.98.3:47569 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 00:20:02 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:20:02 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:20:02.775617 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:02.770646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.1:39095)
0924 00:20:02.770814 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:02.770833 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:02.770919 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:02.771077 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:02.771078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:02.771239 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:02.771288 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:02.771289 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:02.772112 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:02.772113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:02.772792 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:02.772795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:02.773136 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:02.773136 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:02.773173 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:02.773596 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:02.773601 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:02.775557 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":116,"threads_started":1}
W20260924 00:20:02.775936 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:02.770748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39095 (local address 127.0.98.62:35491)
0924 00:20:02.770921 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:02.770925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:02.770943 (+ 18us) server_negotiation.cc:407] Connection header received
0924 00:20:02.770970 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:02.770970 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:02.770982 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:02.771066 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:02.771394 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:02.772026 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:02.772870 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:02.773065 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:02.773131 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:02.775769 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39095: BlockingRecv error: recv got EOF from 127.0.98.1:39095 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":112,"threads_started":1}
Sep 24 00:20:02 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:20:02 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:20:02.906088 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:02.901317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.2:55571)
0924 00:20:02.901452 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:02.901465 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:02.901533 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:02.901703 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:02.901704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:02.901913 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:02.901964 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:02.901966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:02.902647 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:02.902648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:02.903184 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:02.903186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:02.903550 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:02.903551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:02.903607 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:02.904145 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:02.904150 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:02.905988 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
W20260924 00:20:02.906145 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:02.901441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55571 (local address 127.0.98.62:35491)
0924 00:20:02.901498 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:02.901500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:02.901514 (+ 14us) server_negotiation.cc:407] Connection header received
0924 00:20:02.901594 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:02.901595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:02.901610 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:02.901689 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:02.902059 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:02.902563 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:02.903275 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:02.903474 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:02.903553 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:02.906080 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55571: BlockingRecv error: recv got EOF from 127.0.98.2:55571 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 24 00:20:03 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:20:03 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:20:03.045248 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:03.040151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.3:46145)
0924 00:20:03.040378 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:03.040392 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:03.040497 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:03.040655 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:03.040656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:03.040870 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:03.040918 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:03.040920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:03.041586 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:03.041588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:03.042121 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:03.042123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:03.042546 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:03.042547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:03.042605 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:03.043120 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:03.043125 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:03.045141 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":136,"threads_started":1}
W20260924 00:20:03.045339 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:03.040315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46145 (local address 127.0.98.62:35491)
0924 00:20:03.040388 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:03.040391 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:03.040481 (+ 90us) server_negotiation.cc:407] Connection header received
0924 00:20:03.040559 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:03.040560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:03.040575 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:03.040650 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:03.041006 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:03.041502 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:03.042217 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:03.042452 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:03.042544 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:03.045226 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46145: BlockingRecv error: recv got EOF from 127.0.98.3:46145 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 00:20:03 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:20:03 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:20:03.781344 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:03.776205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.1:59069)
0924 00:20:03.776359 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:03.776379 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:03.776443 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:03.776595 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:03.776596 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:03.776828 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:03.776879 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:03.776881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:03.777683 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:03.777684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:03.778241 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:03.778243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:03.778638 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:03.778639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:03.778686 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:03.779223 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:03.779228 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:03.781275 (+ 2047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
W20260924 00:20:03.781566 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:03.776308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59069 (local address 127.0.98.62:35491)
0924 00:20:03.776447 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:03.776448 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:03.776484 (+ 36us) server_negotiation.cc:407] Connection header received
0924 00:20:03.776506 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:03.776507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:03.776518 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:03.776570 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:03.776991 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:03.777577 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:03.778338 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:03.778556 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:03.778620 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:03.781387 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59069: BlockingRecv error: recv got EOF from 127.0.98.1:59069 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
Sep 24 00:20:03 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:20:03 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:20:03.911870 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:03.906627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.2:43003)
0924 00:20:03.906779 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:03.906790 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:03.906857 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:03.907008 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:03.907009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:03.907230 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:03.907281 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:03.907283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:03.908101 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:03.908102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:03.908760 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:03.908763 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:03.909200 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:03.909200 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:03.909249 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:03.909737 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:03.909742 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:03.911743 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
W20260924 00:20:03.911897 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:03.906693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43003 (local address 127.0.98.62:35491)
0924 00:20:03.906790 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:03.906792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:03.906856 (+ 64us) server_negotiation.cc:407] Connection header received
0924 00:20:03.906913 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:03.906914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:03.906926 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:03.906990 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:03.907364 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:03.908006 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:03.908888 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:03.909122 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:03.909189 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:03.911829 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43003: BlockingRecv error: recv got EOF from 127.0.98.2:43003 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 00:20:04 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:20:04 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:20:04.051754 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:04.045999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.3:50001)
0924 00:20:04.046182 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:04.046194 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:04.046277 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:04.046470 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:04.046471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:04.046638 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:04.046687 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:04.046689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:04.047564 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:04.047565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:04.048242 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:04.048245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:04.048647 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:04.048648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:04.048698 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:04.049283 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:04.049288 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:04.051648 (+ 2360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":108,"threads_started":1}
W20260924 00:20:04.051925 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:04.046109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50001 (local address 127.0.98.62:35491)
0924 00:20:04.046209 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:04.046211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:04.046288 (+ 77us) server_negotiation.cc:407] Connection header received
0924 00:20:04.046320 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:04.046321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:04.046334 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:04.046406 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:04.046878 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:04.047471 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:04.048323 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:04.048526 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:04.048596 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:04.051815 (+ 3219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50001: BlockingRecv error: recv got EOF from 127.0.98.3:50001 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 00:20:04 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:20:04 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:20:04.788033 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:04.782211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.1:54283)
0924 00:20:04.782379 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:04.782390 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:04.782455 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:04.782657 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:04.782658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:04.782895 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:04.782957 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:04.782959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:04.783691 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:04.783692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:04.784334 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:04.784337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:04.784935 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:04.784937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:04.784998 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:04.785602 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:04.785608 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:04.787924 (+ 2316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":73,"threads_started":1}
W20260924 00:20:04.788095 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:04.782366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54283 (local address 127.0.98.62:35491)
0924 00:20:04.782504 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:04.782506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:04.782514 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:20:04.782539 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:04.782539 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:04.782549 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:04.782619 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:04.783068 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:04.783591 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:04.784518 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:04.784855 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:04.784933 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:04.788014 (+ 3081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54283: BlockingRecv error: recv got EOF from 127.0.98.1:54283 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
Sep 24 00:20:04 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:20:04 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:20:04.917574 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:04.912515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.2:42981)
0924 00:20:04.912684 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:04.912696 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:04.912784 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:04.912975 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:04.912976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:04.913156 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:04.913203 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:04.913205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:04.913898 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:04.913899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:04.914469 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:04.914471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:04.914866 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:04.914866 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:04.914925 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:04.915578 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:04.915583 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:04.917516 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":62,"threads_started":1}
W20260924 00:20:04.917737 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:04.912690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42981 (local address 127.0.98.62:35491)
0924 00:20:04.912759 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:04.912761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:04.912771 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:20:04.912878 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:04.912879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:04.912890 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:04.912971 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:04.913286 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:04.913804 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:04.914564 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:04.914775 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:04.914865 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:04.917657 (+ 2792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42981: BlockingRecv error: recv got EOF from 127.0.98.2:42981 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 00:20:05 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:20:05 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:20:05.058151 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:05.052571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.3:36877)
0924 00:20:05.052774 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:05.052787 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:05.052875 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:05.053055 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:05.053056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:05.053297 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:05.053350 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:05.053352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:05.054226 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:05.054227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:05.054874 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:05.054877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:05.055261 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:05.055262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:05.055320 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:05.055829 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:05.055834 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:05.058067 (+ 2233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
W20260924 00:20:05.058211 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:05.052662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36877 (local address 127.0.98.62:35491)
0924 00:20:05.052723 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:05.052726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:05.052851 (+ 125us) server_negotiation.cc:407] Connection header received
0924 00:20:05.052959 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:05.052960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:05.052974 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:05.053047 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:05.053470 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:05.054138 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:05.054984 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:05.055180 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:05.055252 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:05.058162 (+ 2910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36877: BlockingRecv error: recv got EOF from 127.0.98.3:36877 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 24 00:20:05 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:20:05 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:20:05.802872 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:05.788658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.1:45815)
0924 00:20:05.788803 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:05.788817 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:05.788890 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:05.789139 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:05.789140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:05.789357 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:05.789411 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:05.789413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:05.790238 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:05.790239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:05.790880 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:05.790883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:05.791493 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:05.791494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:05.791569 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:05.800427 (+ 8858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:05.800433 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:05.802764 (+ 2331us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
W20260924 00:20:05.803038 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:05.788763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45815 (local address 127.0.98.62:35491)
0924 00:20:05.788890 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:05.788892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:05.788903 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:20:05.788935 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:05.788936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:05.788949 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:05.789027 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:05.789525 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:05.790119 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:05.791002 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:05.791398 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:05.791473 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:05.802934 (+ 11461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45815: BlockingRecv error: recv got EOF from 127.0.98.1:45815 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
Sep 24 00:20:05 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:20:05 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:20:05.923103 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:05.918154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.2:45489)
0924 00:20:05.918306 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:05.918323 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:05.918376 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:05.918542 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:05.918542 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:05.918711 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:05.918759 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:05.918760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:05.919516 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:05.919516 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:05.920035 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:05.920037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:05.920473 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:05.920474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:05.920516 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:05.920986 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:05.920990 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:05.922995 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
W20260924 00:20:05.923159 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:05.918231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45489 (local address 127.0.98.62:35491)
0924 00:20:05.918411 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:05.918413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:05.918423 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:20:05.918452 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:05.918452 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:05.918463 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:05.918528 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:05.918851 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:05.919435 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:05.920146 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:05.920403 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:05.920474 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:05.923085 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45489: BlockingRecv error: recv got EOF from 127.0.98.2:45489 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
Sep 24 00:20:06 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:20:06 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:20:06.063841 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:06.058667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.3:36991)
0924 00:20:06.058872 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:06.058886 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:06.058992 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:06.059163 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:06.059164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:06.059368 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:06.059416 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:06.059418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:06.060161 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:06.060162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:06.060792 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:06.060794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:06.061191 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:06.061192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:06.061266 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:06.061810 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:06.061815 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:06.063734 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":87,"threads_started":1}
W20260924 00:20:06.063898 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:06.058770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36991 (local address 127.0.98.62:35491)
0924 00:20:06.058867 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:06.058869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:06.058980 (+ 111us) server_negotiation.cc:407] Connection header received
0924 00:20:06.059068 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:06.059069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:06.059089 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:06.059154 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:06.059505 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:06.060069 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:06.060886 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:06.061096 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:06.061190 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:06.063825 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36991: BlockingRecv error: recv got EOF from 127.0.98.3:36991 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 00:20:06 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:20:06 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:20:06.808538 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:06.803383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.1:49737)
0924 00:20:06.803667 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:06.803679 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:06.803744 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:06.803944 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:06.803945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:06.804117 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:06.804189 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:06.804190 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:06.804917 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:06.804918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:06.805502 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:06.805503 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:06.805901 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:06.805902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:06.805945 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:06.806415 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:06.806419 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:06.808473 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":185,"threads_started":1}
W20260924 00:20:06.808681 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:06.803564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49737 (local address 127.0.98.62:35491)
0924 00:20:06.803680 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:06.803682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:06.803728 (+ 46us) server_negotiation.cc:407] Connection header received
0924 00:20:06.803790 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:06.803791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:06.803800 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:06.803865 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:06.804260 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:06.804786 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:06.805589 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:06.805788 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:06.805845 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:06.808588 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49737: BlockingRecv error: recv got EOF from 127.0.98.1:49737 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":65,"threads_started":1}
Sep 24 00:20:06 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 00:20:06 dist-test-slave-777x krb5kdc[8522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 00:20:06.928956 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:06.923908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35491 (local address 127.0.98.2:40367)
0924 00:20:06.924098 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:06.924110 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:06.924231 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:06.924480 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:06.924481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:06.924714 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:06.924777 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:06.924779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:06.925492 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:06.925493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:06.926065 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:06.926066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:06.926420 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:06.926420 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:06.926463 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:06.926990 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:06.926994 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:06.928877 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35491: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":102,"threads_started":1}
W20260924 00:20:06.929030 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:06.924047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40367 (local address 127.0.98.62:35491)
0924 00:20:06.924122 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:06.924125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:06.924196 (+ 71us) server_negotiation.cc:407] Connection header received
0924 00:20:06.924308 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:06.924308 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:06.924335 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:06.924410 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:06.924871 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:06.925417 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:06.926145 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:06.926349 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:06.926414 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:06.928981 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40367: BlockingRecv error: recv got EOF from 127.0.98.2:40367 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 00:20:06.967481 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 9105
I20260924 00:20:06.972035 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 9257
I20260924 00:20:06.976785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 9390
I20260924 00:20:06.981096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 9035
2026-09-24T00:20:06Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17569 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T00:20:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:20:07Z Disabled control of system clock
I20260924 00:20:07.008733 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38013
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 00:20:07.094777 9585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:20:07.095008 9585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:20:07.095029 9585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:20:07.096207 9585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:20:07.096246 9585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:20:07.096258 9585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:20:07.096271 9585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:20:07.096283 9585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:20:07.097590 9585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38013
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:20:07.097875 9585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:20:07.098127 9585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:07.103006 9590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:07.103065 9591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:07.103091 9593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:07.103673 9585 server_base.cc:1034] running on GCE node
I20260924 00:20:07.103971 9585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:20:07.104319 9585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:20:07.105497 9585 hybrid_clock.cc:648] HybridClock initialized: now 1790209207105482 us; error 28 us; skew 500 ppm
I20260924 00:20:07.106876 9585 webserver.cc:460] Webserver started at http://127.0.98.62:44655/ using document root <none> and password file <none>
I20260924 00:20:07.107125 9585 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:07.107172 9585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:07.107304 9585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:20:07.108526 9585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "472afd3ca43c4836ae924205dac7292a"
format_stamp: "Formatted at 2026-09-24 00:20:07 on dist-test-slave-777x"
server_key: "87ca4740a1e1cf6bd6abce50e3bbaf01"
server_key_iv: "a26f41bb7029bfde29dbdd0411556cf4"
server_key_version: "encryptionkey@0"
I20260924 00:20:07.108888 9585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "472afd3ca43c4836ae924205dac7292a"
format_stamp: "Formatted at 2026-09-24 00:20:07 on dist-test-slave-777x"
server_key: "87ca4740a1e1cf6bd6abce50e3bbaf01"
server_key_iv: "a26f41bb7029bfde29dbdd0411556cf4"
server_key_version: "encryptionkey@0"
I20260924 00:20:07.110224 9585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:20:07.111017 9599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:07.111227 9585 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:20:07.111300 9585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "472afd3ca43c4836ae924205dac7292a"
format_stamp: "Formatted at 2026-09-24 00:20:07 on dist-test-slave-777x"
server_key: "87ca4740a1e1cf6bd6abce50e3bbaf01"
server_key_iv: "a26f41bb7029bfde29dbdd0411556cf4"
server_key_version: "encryptionkey@0"
I20260924 00:20:07.111397 9585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:07.124349 9585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:07.124662 9585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:20:07.124799 9585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:07.129266 9585 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38013
I20260924 00:20:07.129309 9651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38013 every 8 connection(s)
I20260924 00:20:07.129688 9585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:20:07.130429 9652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:07.133234 9652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a: Bootstrap starting.
I20260924 00:20:07.133395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 9585
I20260924 00:20:07.133496 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:20:07.133713 392 external_mini_cluster.cc:1428] Setting key ade06d6a8bcbe541fc81e47ac991852b
I20260924 00:20:07.133998 9652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:07.134328 9652 log.cc:824] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a: Log is configured to *not* fsync() on all Append() calls
I20260924 00:20:07.135078 9652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a: No bootstrap required, opened a new log
I20260924 00:20:07.137456 9652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472afd3ca43c4836ae924205dac7292a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38013 } }
I20260924 00:20:07.137604 9652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:07.137627 9652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 472afd3ca43c4836ae924205dac7292a, State: Initialized, Role: FOLLOWER
I20260924 00:20:07.137777 9652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472afd3ca43c4836ae924205dac7292a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38013 } }
I20260924 00:20:07.137887 9652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:20:07.137921 9652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:20:07.137959 9652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:07.138702 9652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472afd3ca43c4836ae924205dac7292a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38013 } }
I20260924 00:20:07.138844 9652 leader_election.cc:302] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 472afd3ca43c4836ae924205dac7292a; no voters:
I20260924 00:20:07.139046 9652 leader_election.cc:288] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:20:07.139076 9657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:07.139178 9657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a [term 1 LEADER]: Becoming Leader. State: Replica: 472afd3ca43c4836ae924205dac7292a, State: Running, Role: LEADER
I20260924 00:20:07.139333 9652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:20:07.139328 9657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472afd3ca43c4836ae924205dac7292a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38013 } }
I20260924 00:20:07.139786 9658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "472afd3ca43c4836ae924205dac7292a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472afd3ca43c4836ae924205dac7292a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38013 } } }
I20260924 00:20:07.139882 9658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a [sys.catalog]: This master's current role is: LEADER
I20260924 00:20:07.140120 9659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 472afd3ca43c4836ae924205dac7292a. Latest consensus state: current_term: 1 leader_uuid: "472afd3ca43c4836ae924205dac7292a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "472afd3ca43c4836ae924205dac7292a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38013 } } }
I20260924 00:20:07.140283 9659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a [sys.catalog]: This master's current role is: LEADER
I20260924 00:20:07.140316 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:07.134296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59698 (local address 127.0.98.62:38013)
0924 00:20:07.134531 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:07.134541 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:07.134576 (+ 35us) server_negotiation.cc:407] Connection header received
0924 00:20:07.135125 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:07.135135 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:07.135222 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:07.135387 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:07.135580 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:07.137004 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:07.137836 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:07.138456 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:07.138587 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:07.139336 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:20:07.139340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:20:07.139341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:20:07.139548 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:20:07.139588 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:20:07.139602 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:20:07.139641 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:07.139783 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 00:20:07.139864 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":85,"threads_started":1}
W20260924 00:20:07.141484 9673 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:20:07.141547 9673 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:20:07.141607 9664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:20:07.141816 9664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:20:07.143263 9664 catalog_manager.cc:1347] Generated new cluster ID: 52e362dedcb544b39162d6605ea9c92d
I20260924 00:20:07.143317 9664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:20:07.147274 9664 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:20:07.148022 9664 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:20:07.155285 9664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 472afd3ca43c4836ae924205dac7292a: Generated new TSK 0
I20260924 00:20:07.155493 9664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:20:07.164436 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38013
--builtin_ntp_servers=127.0.98.20:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 00:20:07.246716 9676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:20:07.246979 9676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:20:07.247020 9676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:20:07.248234 9676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:20:07.248296 9676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:20:07.248340 9676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:20:07.249701 9676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:20:07.249961 9676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:20:07.250253 9676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:07.256692 9681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:07.256646 9684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:07.256672 9682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:07.256690 9676 server_base.cc:1034] running on GCE node
I20260924 00:20:07.257033 9676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:20:07.257299 9676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:20:07.258472 9676 hybrid_clock.cc:648] HybridClock initialized: now 1790209207258449 us; error 34 us; skew 500 ppm
I20260924 00:20:07.260154 9676 webserver.cc:460] Webserver started at http://127.0.98.1:36229/ using document root <none> and password file <none>
I20260924 00:20:07.260462 9676 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:07.260522 9676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:07.260653 9676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:20:07.262156 9676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "3eb9264985a44774ad3ba95a58cf8bfc"
format_stamp: "Formatted at 2026-09-24 00:20:07 on dist-test-slave-777x"
server_key: "2d0e4ad9c15a2eae2c02e441335c3d22"
server_key_iv: "f0ee6cab736e4bb4964a9e4cc3fd7fec"
server_key_version: "encryptionkey@0"
I20260924 00:20:07.262590 9676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3eb9264985a44774ad3ba95a58cf8bfc"
format_stamp: "Formatted at 2026-09-24 00:20:07 on dist-test-slave-777x"
server_key: "2d0e4ad9c15a2eae2c02e441335c3d22"
server_key_iv: "f0ee6cab736e4bb4964a9e4cc3fd7fec"
server_key_version: "encryptionkey@0"
I20260924 00:20:07.264464 9676 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260924 00:20:07.265394 9690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:07.265618 9676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:20:07.265712 9676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "3eb9264985a44774ad3ba95a58cf8bfc"
format_stamp: "Formatted at 2026-09-24 00:20:07 on dist-test-slave-777x"
server_key: "2d0e4ad9c15a2eae2c02e441335c3d22"
server_key_iv: "f0ee6cab736e4bb4964a9e4cc3fd7fec"
server_key_version: "encryptionkey@0"
I20260924 00:20:07.265787 9676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:07.287446 9676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:07.287750 9676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:20:07.287858 9676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:07.288053 9676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:20:07.288439 9676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:20:07.288475 9676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:07.288498 9676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:20:07.288511 9676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:07.294361 9676 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34509
I20260924 00:20:07.294400 9803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34509 every 8 connection(s)
I20260924 00:20:07.294751 9676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:20:07.299873 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 9676
I20260924 00:20:07.299999 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:20:07.300256 392 external_mini_cluster.cc:1428] Setting key 072460f3eb7004840628ce6b19761708
I20260924 00:20:07.300321 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:07.295510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41219 (local address 127.0.98.62:38013)
0924 00:20:07.295595 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:07.295597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:07.295819 (+ 222us) server_negotiation.cc:407] Connection header received
0924 00:20:07.296411 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:07.296412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:07.296423 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:07.296514 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:07.297256 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:07.297758 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:07.298841 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:07.299039 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:07.299104 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:07.299945 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:20:07.299947 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:20:07.299948 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:20:07.299953 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:20:07.299973 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:20:07.299979 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:20:07.300043 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:07.300191 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 00:20:07.300221 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 00:20:07.300545 9806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:07.295478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38013 (local address 127.0.98.1:41219)
0924 00:20:07.295686 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:07.295725 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:07.296293 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:07.296591 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:07.296594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:07.296602 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:07.297161 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:07.297166 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:07.297848 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:07.297849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:07.298698 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:07.298700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:07.299125 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:07.299126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:07.299243 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:07.299803 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:20:07.299803 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:07.299832 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:20:07.299854 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:20:07.300077 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:20:07.300082 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:20:07.300086 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:20:07.300151 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 00:20:07.300281 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":104,"threads_started":1}
I20260924 00:20:07.301169 9804 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38013
I20260924 00:20:07.301262 9804 heartbeater.cc:464] Registering TS with master...
I20260924 00:20:07.301432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38013
--builtin_ntp_servers=127.0.98.20:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 00:20:07.301460 9804 heartbeater.cc:511] Master 127.0.98.62:38013 requested a full tablet report, sending...
I20260924 00:20:07.301955 9616 ts_manager.cc:194] Registered new tserver with Master: 3eb9264985a44774ad3ba95a58cf8bfc (127.0.98.1:34509)
I20260924 00:20:07.302760 9616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41219
W20260924 00:20:07.386021 9807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:20:07.386228 9807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:20:07.386251 9807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:20:07.387454 9807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:20:07.387496 9807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:20:07.387521 9807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:20:07.388947 9807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:20:07.389195 9807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:20:07.389508 9807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:07.394718 9813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:07.394730 9815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:07.395056 9807 server_base.cc:1034] running on GCE node
W20260924 00:20:07.395115 9812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:07.395371 9807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:20:07.395644 9807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:20:07.396809 9807 hybrid_clock.cc:648] HybridClock initialized: now 1790209207396791 us; error 63 us; skew 500 ppm
I20260924 00:20:07.398213 9807 webserver.cc:460] Webserver started at http://127.0.98.2:36775/ using document root <none> and password file <none>
I20260924 00:20:07.398470 9807 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:07.398537 9807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:07.398679 9807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:20:07.399765 9807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "8e9d82d98f3e4adba7f825ed97b5ba82"
format_stamp: "Formatted at 2026-09-24 00:20:07 on dist-test-slave-777x"
server_key: "7517e777a31e06e78bf6bf7c86a45830"
server_key_iv: "6120cd32bc31f715c53ab173b5cdacec"
server_key_version: "encryptionkey@0"
I20260924 00:20:07.400133 9807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8e9d82d98f3e4adba7f825ed97b5ba82"
format_stamp: "Formatted at 2026-09-24 00:20:07 on dist-test-slave-777x"
server_key: "7517e777a31e06e78bf6bf7c86a45830"
server_key_iv: "6120cd32bc31f715c53ab173b5cdacec"
server_key_version: "encryptionkey@0"
I20260924 00:20:07.401490 9807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:20:07.402343 9821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:07.402522 9807 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:20:07.402622 9807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "8e9d82d98f3e4adba7f825ed97b5ba82"
format_stamp: "Formatted at 2026-09-24 00:20:07 on dist-test-slave-777x"
server_key: "7517e777a31e06e78bf6bf7c86a45830"
server_key_iv: "6120cd32bc31f715c53ab173b5cdacec"
server_key_version: "encryptionkey@0"
I20260924 00:20:07.402694 9807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:07.411880 9807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:07.412222 9807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:20:07.412367 9807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:07.412578 9807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:20:07.412919 9807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:20:07.412973 9807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:07.413013 9807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:20:07.413039 9807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:07.419221 9807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37863
I20260924 00:20:07.419282 9934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37863 every 8 connection(s)
I20260924 00:20:07.419617 9807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:20:07.425267 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:07.420470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44689 (local address 127.0.98.62:38013)
0924 00:20:07.420531 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:07.420533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:07.420803 (+ 270us) server_negotiation.cc:407] Connection header received
0924 00:20:07.421416 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:07.421418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:07.421430 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:07.421488 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:07.422176 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:07.422707 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:07.423713 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:07.423914 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:07.423988 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:07.424970 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:20:07.424971 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:20:07.424972 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:20:07.424977 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:20:07.424993 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:20:07.424997 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:20:07.425060 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:07.425158 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 00:20:07.425186 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 00:20:07.425494 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:07.420481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38013 (local address 127.0.98.2:44689)
0924 00:20:07.420682 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:07.420716 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:07.421305 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:07.421568 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:07.421571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:07.421580 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:07.422082 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:07.422086 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:07.422814 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:07.422820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:07.423612 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:07.423614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:07.423993 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:07.423994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:07.424128 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:07.424721 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:20:07.424722 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:07.424752 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:20:07.424774 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:20:07.425073 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:20:07.425079 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:20:07.425082 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:20:07.425162 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 00:20:07.425268 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260924 00:20:07.426051 9935 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38013
I20260924 00:20:07.426136 9935 heartbeater.cc:464] Registering TS with master...
I20260924 00:20:07.426281 9935 heartbeater.cc:511] Master 127.0.98.62:38013 requested a full tablet report, sending...
I20260924 00:20:07.426431 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 9807
I20260924 00:20:07.426528 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:20:07.426658 9616 ts_manager.cc:194] Registered new tserver with Master: 8e9d82d98f3e4adba7f825ed97b5ba82 (127.0.98.2:37863)
I20260924 00:20:07.426723 392 external_mini_cluster.cc:1428] Setting key 5f3dcd5d89342ccda1dc9556ac8e721a
I20260924 00:20:07.427204 9616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44689
I20260924 00:20:07.427853 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38013
--builtin_ntp_servers=127.0.98.20:42307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 00:20:07.512362 9938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:20:07.512596 9938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:20:07.512629 9938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:20:07.513886 9938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:20:07.513945 9938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:20:07.513986 9938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:20:07.515413 9938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:20:07.515690 9938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:20:07.515996 9938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:07.521180 9946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:07.521152 9943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:07.521349 9938 server_base.cc:1034] running on GCE node
W20260924 00:20:07.521461 9944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:07.521677 9938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:20:07.522037 9938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:20:07.523200 9938 hybrid_clock.cc:648] HybridClock initialized: now 1790209207523191 us; error 30 us; skew 500 ppm
I20260924 00:20:07.524587 9938 webserver.cc:460] Webserver started at http://127.0.98.3:39999/ using document root <none> and password file <none>
I20260924 00:20:07.524906 9938 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:07.524960 9938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:07.525098 9938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:20:07.526203 9938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "bac73cef484249afa9561eefcc1ca776"
format_stamp: "Formatted at 2026-09-24 00:20:07 on dist-test-slave-777x"
server_key: "7d6dba7e85f0902ba59c30f19c56aaac"
server_key_iv: "589e10fdf096f4e0c2653a89d29ab8b5"
server_key_version: "encryptionkey@0"
I20260924 00:20:07.526597 9938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bac73cef484249afa9561eefcc1ca776"
format_stamp: "Formatted at 2026-09-24 00:20:07 on dist-test-slave-777x"
server_key: "7d6dba7e85f0902ba59c30f19c56aaac"
server_key_iv: "589e10fdf096f4e0c2653a89d29ab8b5"
server_key_version: "encryptionkey@0"
I20260924 00:20:07.527913 9938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:20:07.528803 9952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:07.529011 9938 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:20:07.529116 9938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "bac73cef484249afa9561eefcc1ca776"
format_stamp: "Formatted at 2026-09-24 00:20:07 on dist-test-slave-777x"
server_key: "7d6dba7e85f0902ba59c30f19c56aaac"
server_key_iv: "589e10fdf096f4e0c2653a89d29ab8b5"
server_key_version: "encryptionkey@0"
I20260924 00:20:07.529191 9938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:07.540855 9938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:07.541211 9938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:20:07.541366 9938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:07.541626 9938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:20:07.541988 9938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:20:07.542042 9938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:07.542088 9938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:20:07.542114 9938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:07.548396 9938 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36659
I20260924 00:20:07.548437 10065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36659 every 8 connection(s)
I20260924 00:20:07.548803 9938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:20:07.552781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 9938
I20260924 00:20:07.552889 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:20:07.553076 392 external_mini_cluster.cc:1428] Setting key 57479054afdaba018fb61adbb67c8086
I20260924 00:20:07.554524 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:07.549601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35173 (local address 127.0.98.62:38013)
0924 00:20:07.549665 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:07.549667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:07.549916 (+ 249us) server_negotiation.cc:407] Connection header received
0924 00:20:07.550414 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:07.550416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:07.550425 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:07.550497 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:07.551149 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:07.551634 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:07.552780 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:07.553124 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:07.553202 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:07.554266 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:20:07.554268 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:20:07.554268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:20:07.554274 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:20:07.554290 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:20:07.554295 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:20:07.554339 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:07.554416 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 00:20:07.554444 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 00:20:07.554760 10068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:07.549585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38013 (local address 127.0.98.3:35173)
0924 00:20:07.549784 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:07.549829 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:07.550332 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:07.550556 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:07.550560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:07.550568 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:07.551080 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:07.551084 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:07.551721 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:07.551722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:07.552696 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:07.552698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:07.553209 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:07.553210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:07.553418 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:07.554115 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:20:07.554116 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:07.554161 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:20:07.554185 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:20:07.554350 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:20:07.554355 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:20:07.554357 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:20:07.554421 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 00:20:07.554551 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260924 00:20:07.555168 10066 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38013
I20260924 00:20:07.555262 10066 heartbeater.cc:464] Registering TS with master...
I20260924 00:20:07.555408 10066 heartbeater.cc:511] Master 127.0.98.62:38013 requested a full tablet report, sending...
I20260924 00:20:07.555775 9616 ts_manager.cc:194] Registered new tserver with Master: bac73cef484249afa9561eefcc1ca776 (127.0.98.3:36659)
I20260924 00:20:07.556238 9616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:35173
I20260924 00:20:07.565253 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 00:20:07.567243 9655 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:07.566586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59714 (local address 127.0.98.62:38013)
0924 00:20:07.566630 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:07.566633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:07.566748 (+ 115us) server_negotiation.cc:407] Connection header received
0924 00:20:07.566908 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:07.566910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:07.566921 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:07.566995 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:07.567152 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59714: BlockingRecv error: recv got EOF from 127.0.0.1:59714 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:20:07.567291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 9676
I20260924 00:20:07.571924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 9807
I20260924 00:20:07.576377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 9938
I20260924 00:20:07.580603 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 9585
2026-09-24T00:20:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (598 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T00:20:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:20:07Z Disabled control of system clock
I20260924 00:20:07.607267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38849
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 00:20:07.690693 10077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:20:07.690914 10077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:20:07.690936 10077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:20:07.692145 10077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:20:07.692255 10077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:20:07.692271 10077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:20:07.692284 10077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:20:07.692296 10077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:20:07.693635 10077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38849
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:20:07.693840 10077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:20:07.694099 10077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:07.699218 10082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:07.699120 10083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:07.699164 10077 server_base.cc:1034] running on GCE node
W20260924 00:20:07.699123 10085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:07.699736 10077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:20:07.700057 10077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:20:07.701203 10077 hybrid_clock.cc:648] HybridClock initialized: now 1790209207701195 us; error 53 us; skew 500 ppm
I20260924 00:20:07.702490 10077 webserver.cc:460] Webserver started at http://127.0.98.62:42137/ using document root <none> and password file <none>
I20260924 00:20:07.702744 10077 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:07.702812 10077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:07.702981 10077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:20:07.704092 10077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "327235dbb8c4434b8a41c49de3e2a8e0"
format_stamp: "Formatted at 2026-09-24 00:20:07 on dist-test-slave-777x"
server_key: "ad61e4100e672d1b6c2de495f4cb283a"
server_key_iv: "6a60ee8b34c91639bbd5ef967d0ccbee"
server_key_version: "encryptionkey@0"
I20260924 00:20:07.704520 10077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "327235dbb8c4434b8a41c49de3e2a8e0"
format_stamp: "Formatted at 2026-09-24 00:20:07 on dist-test-slave-777x"
server_key: "ad61e4100e672d1b6c2de495f4cb283a"
server_key_iv: "6a60ee8b34c91639bbd5ef967d0ccbee"
server_key_version: "encryptionkey@0"
I20260924 00:20:07.705861 10077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:20:07.706681 10091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:07.706880 10077 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:20:07.707012 10077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "327235dbb8c4434b8a41c49de3e2a8e0"
format_stamp: "Formatted at 2026-09-24 00:20:07 on dist-test-slave-777x"
server_key: "ad61e4100e672d1b6c2de495f4cb283a"
server_key_iv: "6a60ee8b34c91639bbd5ef967d0ccbee"
server_key_version: "encryptionkey@0"
I20260924 00:20:07.707088 10077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:07.710911 10077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:07.711200 10077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:20:07.711344 10077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:07.715240 10077 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38849
I20260924 00:20:07.715271 10143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38849 every 8 connection(s)
I20260924 00:20:07.715602 10077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:20:07.716305 10144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:07.719398 10144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0: Bootstrap starting.
I20260924 00:20:07.720090 10144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:07.720439 10144 log.cc:824] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0: Log is configured to *not* fsync() on all Append() calls
I20260924 00:20:07.721151 10144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0: No bootstrap required, opened a new log
I20260924 00:20:07.721662 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 10077
I20260924 00:20:07.721741 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:20:07.721910 392 external_mini_cluster.cc:1428] Setting key 874bce3a244d07314607cebfdee10210
I20260924 00:20:07.723491 10144 raft_consensus.cc:348] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327235dbb8c4434b8a41c49de3e2a8e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38849 } }
I20260924 00:20:07.723673 10144 raft_consensus.cc:374] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:07.723709 10144 raft_consensus.cc:729] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327235dbb8c4434b8a41c49de3e2a8e0, State: Initialized, Role: FOLLOWER
I20260924 00:20:07.723877 10144 consensus_queue.cc:260] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327235dbb8c4434b8a41c49de3e2a8e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38849 } }
I20260924 00:20:07.723978 10144 raft_consensus.cc:388] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:20:07.724011 10144 raft_consensus.cc:482] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:20:07.724056 10144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:07.724858 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:07.722431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54486 (local address 127.0.98.62:38849)
0924 00:20:07.722669 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:07.722680 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:07.722714 (+ 34us) server_negotiation.cc:407] Connection header received
0924 00:20:07.723440 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:07.723449 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:07.723502 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:07.723660 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:07.724364 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:20:07.724367 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:20:07.724368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:20:07.724372 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:20:07.724420 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:20:07.724433 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:20:07.724508 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:07.724562 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 00:20:07.724624 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":105,"threads_started":1}
I20260924 00:20:07.725189 10144 raft_consensus.cc:504] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327235dbb8c4434b8a41c49de3e2a8e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38849 } }
I20260924 00:20:07.725294 10144 leader_election.cc:302] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 327235dbb8c4434b8a41c49de3e2a8e0; no voters:
I20260924 00:20:07.725443 10144 leader_election.cc:288] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:20:07.725524 10149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:07.725661 10149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0 [term 1 LEADER]: Becoming Leader. State: Replica: 327235dbb8c4434b8a41c49de3e2a8e0, State: Running, Role: LEADER
I20260924 00:20:07.725703 10144 sys_catalog.cc:564] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:20:07.725755 10149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327235dbb8c4434b8a41c49de3e2a8e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38849 } }
I20260924 00:20:07.726086 10150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "327235dbb8c4434b8a41c49de3e2a8e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327235dbb8c4434b8a41c49de3e2a8e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38849 } } }
I20260924 00:20:07.726080 10151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 327235dbb8c4434b8a41c49de3e2a8e0. Latest consensus state: current_term: 1 leader_uuid: "327235dbb8c4434b8a41c49de3e2a8e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "327235dbb8c4434b8a41c49de3e2a8e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38849 } } }
I20260924 00:20:07.726226 10150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0 [sys.catalog]: This master's current role is: LEADER
I20260924 00:20:07.726233 10151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0 [sys.catalog]: This master's current role is: LEADER
I20260924 00:20:07.726467 10158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:20:07.726970 10158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:20:07.728401 10158 catalog_manager.cc:1347] Generated new cluster ID: b895b3e216bc403e9cf23102a558b1f2
I20260924 00:20:07.728459 10158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:20:07.758190 10158 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:20:07.758392 10158 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:20:07.768392 10158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 327235dbb8c4434b8a41c49de3e2a8e0: Generated new TSK 0
I20260924 00:20:07.768610 10158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:20:07.772140 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38849
--builtin_ntp_servers=127.0.98.20:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 00:20:07.861685 10168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:20:07.861932 10168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:20:07.861977 10168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:20:07.863128 10168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:20:07.863188 10168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:20:07.863231 10168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:20:07.864635 10168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:20:07.864890 10168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:20:07.865186 10168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:07.870879 10173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:07.870873 10174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:07.871150 10176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:07.871155 10168 server_base.cc:1034] running on GCE node
I20260924 00:20:07.871403 10168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:20:07.871649 10168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:20:07.872824 10168 hybrid_clock.cc:648] HybridClock initialized: now 1790209207872804 us; error 33 us; skew 500 ppm
I20260924 00:20:07.874353 10168 webserver.cc:460] Webserver started at http://127.0.98.1:33463/ using document root <none> and password file <none>
I20260924 00:20:07.874615 10168 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:07.874673 10168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:07.874797 10168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:20:07.876209 10168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "c3920b260da345458d8be0284f50bf3a"
format_stamp: "Formatted at 2026-09-24 00:20:07 on dist-test-slave-777x"
server_key: "e39a1f6701a71febb74e2f7f0b2fbc5d"
server_key_iv: "ebf6474831e38e24bc79c7cff2e12a39"
server_key_version: "encryptionkey@0"
I20260924 00:20:07.876646 10168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c3920b260da345458d8be0284f50bf3a"
format_stamp: "Formatted at 2026-09-24 00:20:07 on dist-test-slave-777x"
server_key: "e39a1f6701a71febb74e2f7f0b2fbc5d"
server_key_iv: "ebf6474831e38e24bc79c7cff2e12a39"
server_key_version: "encryptionkey@0"
I20260924 00:20:07.878067 10168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:20:07.878782 10182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:07.878965 10168 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:20:07.879045 10168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "c3920b260da345458d8be0284f50bf3a"
format_stamp: "Formatted at 2026-09-24 00:20:07 on dist-test-slave-777x"
server_key: "e39a1f6701a71febb74e2f7f0b2fbc5d"
server_key_iv: "ebf6474831e38e24bc79c7cff2e12a39"
server_key_version: "encryptionkey@0"
I20260924 00:20:07.879097 10168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:07.882432 10168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:07.882659 10168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:20:07.882755 10168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:07.882908 10168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:20:07.883172 10168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:20:07.883200 10168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:07.883219 10168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:20:07.883232 10168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:07.888780 10168 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33955
I20260924 00:20:07.888947 10295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33955 every 8 connection(s)
I20260924 00:20:07.889097 10168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:20:07.892102 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:07.889813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54831 (local address 127.0.98.62:38849)
0924 00:20:07.889878 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:07.889880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:07.890116 (+ 236us) server_negotiation.cc:407] Connection header received
0924 00:20:07.890605 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:07.890607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:07.890613 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:07.890713 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:07.891853 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:20:07.891854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:20:07.891855 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:20:07.891860 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:20:07.891878 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:20:07.891883 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:20:07.891941 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:07.892022 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 00:20:07.892036 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 00:20:07.892320 10298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:07.889789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38849 (local address 127.0.98.1:54831)
0924 00:20:07.889994 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:07.890028 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:07.890500 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:07.890782 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:07.890786 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:07.890794 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:07.891704 (+ 910us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:20:07.891705 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:07.891732 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:20:07.891760 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:20:07.891947 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:20:07.891952 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:20:07.891954 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:20:07.892016 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 00:20:07.892065 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":100,"threads_started":1}
I20260924 00:20:07.892920 10296 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38849
I20260924 00:20:07.893005 10296 heartbeater.cc:464] Registering TS with master...
I20260924 00:20:07.893127 10296 heartbeater.cc:511] Master 127.0.98.62:38849 requested a full tablet report, sending...
I20260924 00:20:07.893644 10108 ts_manager.cc:194] Registered new tserver with Master: c3920b260da345458d8be0284f50bf3a (127.0.98.1:33955)
I20260924 00:20:07.897619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 10168
I20260924 00:20:07.897701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:20:07.897852 392 external_mini_cluster.cc:1428] Setting key c9b0354d2b8d35c19d64055521059677
I20260924 00:20:07.898780 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38849
--builtin_ntp_servers=127.0.98.20:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 00:20:07.978845 10299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:20:07.979107 10299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:20:07.979146 10299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:20:07.980345 10299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:20:07.980413 10299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:20:07.980455 10299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:20:07.981897 10299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:20:07.982167 10299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:20:07.982467 10299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:07.987445 10305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:07.987579 10307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:07.987675 10304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:07.987816 10299 server_base.cc:1034] running on GCE node
I20260924 00:20:07.988006 10299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:20:07.988279 10299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:20:07.989449 10299 hybrid_clock.cc:648] HybridClock initialized: now 1790209207989434 us; error 27 us; skew 500 ppm
I20260924 00:20:07.990810 10299 webserver.cc:460] Webserver started at http://127.0.98.2:43149/ using document root <none> and password file <none>
I20260924 00:20:07.991065 10299 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:07.991113 10299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:07.991274 10299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:20:07.992461 10299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "102590a313af423f8cbde0145aea103a"
format_stamp: "Formatted at 2026-09-24 00:20:07 on dist-test-slave-777x"
server_key: "2b704a8d263d56d673402ae4c694c473"
server_key_iv: "7be2f08d2f56871cb5bcf7980c949132"
server_key_version: "encryptionkey@0"
I20260924 00:20:07.992821 10299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "102590a313af423f8cbde0145aea103a"
format_stamp: "Formatted at 2026-09-24 00:20:07 on dist-test-slave-777x"
server_key: "2b704a8d263d56d673402ae4c694c473"
server_key_iv: "7be2f08d2f56871cb5bcf7980c949132"
server_key_version: "encryptionkey@0"
I20260924 00:20:07.994149 10299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:20:07.994890 10313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:07.995097 10299 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:20:07.995186 10299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "102590a313af423f8cbde0145aea103a"
format_stamp: "Formatted at 2026-09-24 00:20:07 on dist-test-slave-777x"
server_key: "2b704a8d263d56d673402ae4c694c473"
server_key_iv: "7be2f08d2f56871cb5bcf7980c949132"
server_key_version: "encryptionkey@0"
I20260924 00:20:07.995256 10299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:07.998759 10299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:07.999027 10299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:20:07.999161 10299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:07.999357 10299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:20:07.999668 10299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:20:07.999702 10299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:07.999738 10299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:20:07.999765 10299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:08.005402 10299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32957
I20260924 00:20:08.005442 10426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32957 every 8 connection(s)
I20260924 00:20:08.005810 10299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:20:08.009037 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:08.006573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42953 (local address 127.0.98.62:38849)
0924 00:20:08.006637 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:08.006639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:08.006850 (+ 211us) server_negotiation.cc:407] Connection header received
0924 00:20:08.007354 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:08.007355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:08.007361 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:08.007423 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:08.008723 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:20:08.008724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:20:08.008725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:20:08.008729 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:20:08.008743 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:20:08.008748 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:20:08.008869 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:08.008964 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 00:20:08.008977 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 00:20:08.009230 10429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:08.006566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38849 (local address 127.0.98.2:42953)
0924 00:20:08.006735 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:08.006768 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:08.007222 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:08.007508 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:08.007512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:08.007521 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:08.008565 (+ 1044us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:20:08.008566 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:08.008595 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:20:08.008625 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:20:08.008886 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:20:08.008890 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:20:08.008893 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:20:08.008958 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 00:20:08.009008 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260924 00:20:08.009792 10427 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38849
I20260924 00:20:08.009948 10427 heartbeater.cc:464] Registering TS with master...
I20260924 00:20:08.010123 10427 heartbeater.cc:511] Master 127.0.98.62:38849 requested a full tablet report, sending...
I20260924 00:20:08.010507 10108 ts_manager.cc:194] Registered new tserver with Master: 102590a313af423f8cbde0145aea103a (127.0.98.2:32957)
I20260924 00:20:08.013695 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 10299
I20260924 00:20:08.013896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:20:08.014117 392 external_mini_cluster.cc:1428] Setting key 015a60a70c177cfc596a00ceecbeee59
I20260924 00:20:08.015159 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38849
--builtin_ntp_servers=127.0.98.20:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 00:20:08.096503 10430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:20:08.096700 10430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:20:08.096724 10430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:20:08.097965 10430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:20:08.098004 10430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:20:08.098030 10430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:20:08.099447 10430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:20:08.099685 10430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:20:08.099977 10430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:08.105036 10438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:08.105126 10430 server_base.cc:1034] running on GCE node
W20260924 00:20:08.105036 10435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:08.105132 10436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:08.105577 10430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:20:08.105854 10430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:20:08.107009 10430 hybrid_clock.cc:648] HybridClock initialized: now 1790209208107002 us; error 34 us; skew 500 ppm
I20260924 00:20:08.108289 10430 webserver.cc:460] Webserver started at http://127.0.98.3:34947/ using document root <none> and password file <none>
I20260924 00:20:08.108536 10430 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:08.108603 10430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:08.108736 10430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:20:08.109972 10430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "27062805b95a4f6aa2057185252b0046"
format_stamp: "Formatted at 2026-09-24 00:20:08 on dist-test-slave-777x"
server_key: "5b59a9bf9613f5359dff43751cd2ecfc"
server_key_iv: "2e5990b77e7504081f253c579935f372"
server_key_version: "encryptionkey@0"
I20260924 00:20:08.110333 10430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "27062805b95a4f6aa2057185252b0046"
format_stamp: "Formatted at 2026-09-24 00:20:08 on dist-test-slave-777x"
server_key: "5b59a9bf9613f5359dff43751cd2ecfc"
server_key_iv: "2e5990b77e7504081f253c579935f372"
server_key_version: "encryptionkey@0"
I20260924 00:20:08.111670 10430 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:20:08.112452 10444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:08.112602 10430 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 00:20:08.112686 10430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "27062805b95a4f6aa2057185252b0046"
format_stamp: "Formatted at 2026-09-24 00:20:08 on dist-test-slave-777x"
server_key: "5b59a9bf9613f5359dff43751cd2ecfc"
server_key_iv: "2e5990b77e7504081f253c579935f372"
server_key_version: "encryptionkey@0"
I20260924 00:20:08.112759 10430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:08.116119 10430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:08.116430 10430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:20:08.116576 10430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:08.116779 10430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:20:08.117089 10430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:20:08.117137 10430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:08.117177 10430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:20:08.117205 10430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:08.123206 10430 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40187
I20260924 00:20:08.123245 10557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40187 every 8 connection(s)
I20260924 00:20:08.123601 10430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:20:08.126924 10148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:08.124400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41477 (local address 127.0.98.62:38849)
0924 00:20:08.124462 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:08.124464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:08.124722 (+ 258us) server_negotiation.cc:407] Connection header received
0924 00:20:08.125271 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:08.125273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:08.125281 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:08.125390 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:08.126726 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:20:08.126728 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:20:08.126728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:20:08.126732 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:20:08.126747 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:20:08.126751 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:20:08.126800 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:08.126856 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 00:20:08.126868 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 00:20:08.127137 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:08.124410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38849 (local address 127.0.98.3:41477)
0924 00:20:08.124605 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:08.124639 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:08.125098 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:08.125464 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:08.125483 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:08.125492 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:08.126591 (+ 1099us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:20:08.126592 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:08.126622 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:20:08.126651 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:20:08.126804 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:20:08.126807 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:20:08.126809 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:20:08.126857 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 00:20:08.126905 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
I20260924 00:20:08.127872 10558 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38849
I20260924 00:20:08.127964 10558 heartbeater.cc:464] Registering TS with master...
I20260924 00:20:08.128125 10558 heartbeater.cc:511] Master 127.0.98.62:38849 requested a full tablet report, sending...
I20260924 00:20:08.128605 10108 ts_manager.cc:194] Registered new tserver with Master: 27062805b95a4f6aa2057185252b0046 (127.0.98.3:40187)
I20260924 00:20:08.129959 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 10430
I20260924 00:20:08.130084 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:20:08.130300 392 external_mini_cluster.cc:1428] Setting key 71738395bc39df1fb7d5695f36f8c6d6
I20260924 00:20:08.131191 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 00:20:08.133054 10148 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:20:08.132624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54502 (local address 127.0.98.62:38849)
0924 00:20:08.132681 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:08.132683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:08.132767 (+ 84us) server_negotiation.cc:407] Connection header received
0924 00:20:08.132818 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:08.132819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:08.132823 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:08.132887 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:08.133002 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54502: BlockingRecv error: recv got EOF from 127.0.0.1:54502 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 00:20:08.133176 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 10168
I20260924 00:20:08.138195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 10299
I20260924 00:20:08.142920 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 10430
I20260924 00:20:08.147033 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 10077
2026-09-24T00:20:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (566 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:20:08 dist-test-slave-777x krb5kdc[10568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:20:08 dist-test-slave-777x krb5kdc[10568](info): set up 2 sockets
Sep 24 00:20:08 dist-test-slave-777x krb5kdc[10568](info): commencing operation
krb5kdc: starting...
W20260924 00:20:09.241329 392 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:20:09 dist-test-slave-777x krb5kdc[10568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:20:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:20:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:20:09.366729 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37263
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:20:09.449615 10581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:20:09.449910 10581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:20:09.449952 10581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:20:09.451138 10581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:20:09.451202 10581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:20:09.451226 10581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:20:09.451248 10581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:20:09.451277 10581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:20:09.452898 10581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:20:09.453184 10581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:20:09.453411 10581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:09.458554 10586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:09.458566 10587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:09.458566 10589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:09.458599 10581 server_base.cc:1034] running on GCE node
I20260924 00:20:09.459118 10581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:20:09.459434 10581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:20:09.460580 10581 hybrid_clock.cc:648] HybridClock initialized: now 1790209209460572 us; error 35 us; skew 500 ppm
Sep 24 00:20:09 dist-test-slave-777x krb5kdc[10568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:20:09.463107 10581 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:20:09.463573 10581 webserver.cc:460] Webserver started at http://127.0.98.62:40683/ using document root <none> and password file <none>
I20260924 00:20:09.463855 10581 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:09.463913 10581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:09.464051 10581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:20:09.465159 10581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "d4102f9cbb274166ac3676fee97673b3"
format_stamp: "Formatted at 2026-09-24 00:20:09 on dist-test-slave-777x"
server_key: "5a62bd46272bc99bd289eb29a116d942"
server_key_iv: "6b8e1ae3eb51af495da1556f19d4f62e"
server_key_version: "encryptionkey@0"
I20260924 00:20:09.465504 10581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "d4102f9cbb274166ac3676fee97673b3"
format_stamp: "Formatted at 2026-09-24 00:20:09 on dist-test-slave-777x"
server_key: "5a62bd46272bc99bd289eb29a116d942"
server_key_iv: "6b8e1ae3eb51af495da1556f19d4f62e"
server_key_version: "encryptionkey@0"
I20260924 00:20:09.466871 10581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:20:09.467921 10596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:09.468250 10581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:20:09.468326 10581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "d4102f9cbb274166ac3676fee97673b3"
format_stamp: "Formatted at 2026-09-24 00:20:09 on dist-test-slave-777x"
server_key: "5a62bd46272bc99bd289eb29a116d942"
server_key_iv: "6b8e1ae3eb51af495da1556f19d4f62e"
server_key_version: "encryptionkey@0"
I20260924 00:20:09.468444 10581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:09.494344 10581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:09.496758 10581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:20:09.496901 10581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:09.501128 10581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37263
I20260924 00:20:09.501164 10648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37263 every 8 connection(s)
I20260924 00:20:09.501534 10581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:20:09.502348 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 10581
I20260924 00:20:09.502493 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:20:09.502694 392 external_mini_cluster.cc:1428] Setting key 7048976c0d01e3b1f8a3c1038b3cf368
I20260924 00:20:09.502750 10649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:09.506069 10649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3: Bootstrap starting.
I20260924 00:20:09.506744 10649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:09.507042 10649 log.cc:824] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3: Log is configured to *not* fsync() on all Append() calls
I20260924 00:20:09.507833 10649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3: No bootstrap required, opened a new log
Sep 24 00:20:09 dist-test-slave-777x krb5kdc[10568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:20:09.510087 10649 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4102f9cbb274166ac3676fee97673b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37263 } }
I20260924 00:20:09.510270 10649 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:09.510324 10649 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4102f9cbb274166ac3676fee97673b3, State: Initialized, Role: FOLLOWER
I20260924 00:20:09.510452 10649 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4102f9cbb274166ac3676fee97673b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37263 } }
I20260924 00:20:09.510514 10649 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:20:09.510536 10649 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:20:09.510609 10649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:09.511284 10649 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4102f9cbb274166ac3676fee97673b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37263 } }
I20260924 00:20:09.511373 10649 leader_election.cc:302] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4102f9cbb274166ac3676fee97673b3; no voters:
I20260924 00:20:09.511561 10649 leader_election.cc:288] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:20:09.511591 10654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:09.511698 10654 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3 [term 1 LEADER]: Becoming Leader. State: Replica: d4102f9cbb274166ac3676fee97673b3, State: Running, Role: LEADER
I20260924 00:20:09.511778 10654 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4102f9cbb274166ac3676fee97673b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37263 } }
I20260924 00:20:09.511832 10649 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:20:09.512104 10654 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4102f9cbb274166ac3676fee97673b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4102f9cbb274166ac3676fee97673b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37263 } } }
I20260924 00:20:09.512229 10654 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3 [sys.catalog]: This master's current role is: LEADER
I20260924 00:20:09.512349 10654 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4102f9cbb274166ac3676fee97673b3. Latest consensus state: current_term: 1 leader_uuid: "d4102f9cbb274166ac3676fee97673b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4102f9cbb274166ac3676fee97673b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37263 } } }
I20260924 00:20:09.512434 10654 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3 [sys.catalog]: This master's current role is: LEADER
I20260924 00:20:09.512657 10659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:20:09.513379 10659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:20:09.514220 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:09.503193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33532 (local address 127.0.98.62:37263)
0924 00:20:09.503361 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:09.503368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:09.503393 (+ 25us) server_negotiation.cc:407] Connection header received
0924 00:20:09.503857 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:09.503864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:09.503911 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:09.504031 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:09.504802 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:09.505678 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:09.506687 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:09.507166 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:09.507268 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:09.509837 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:20:09.509842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:20:09.509842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:20:09.509853 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:20:09.511934 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:20:09.512636 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:20:09.512639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:20:09.512639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:20:09.512741 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:20:09.513052 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:20:09.513053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:20:09.513053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:20:09.513372 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:20:09.513506 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:09.513841 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 00:20:09.514017 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260924 00:20:09.515264 10659 catalog_manager.cc:1347] Generated new cluster ID: 9adf3c04ad744f86b79d175c93e43b3b
I20260924 00:20:09.515306 10659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:20:09.521507 10659 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:20:09.522087 10659 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:20:09.527304 10659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3: Generated new TSK 0
I20260924 00:20:09.527495 10659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:20:09.560055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37263
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:20:09.643126 10675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:20:09.643349 10675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:20:09.643375 10675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:20:09.644701 10675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:20:09.644748 10675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:20:09.644775 10675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:20:09.646318 10675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:20:09.646597 10675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:20:09.646838 10675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:09.652029 10680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:09.652137 10675 server_base.cc:1034] running on GCE node
W20260924 00:20:09.652148 10683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:09.652235 10681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:09.652545 10675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:20:09.652868 10675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:20:09.654034 10675 hybrid_clock.cc:648] HybridClock initialized: now 1790209209653995 us; error 58 us; skew 500 ppm
Sep 24 00:20:09 dist-test-slave-777x krb5kdc[10568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:20:09.656991 10675 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:20:09.657488 10675 webserver.cc:460] Webserver started at http://127.0.98.1:36271/ using document root <none> and password file <none>
I20260924 00:20:09.657745 10675 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:09.657814 10675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:09.657979 10675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:20:09.659147 10675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "58fd7c07e2a34ced99835a57dd970ccf"
format_stamp: "Formatted at 2026-09-24 00:20:09 on dist-test-slave-777x"
server_key: "51463f6bbf322f2daa8cd2fbf0f2a287"
server_key_iv: "b1fa8da43033072447e8c91397dffe09"
server_key_version: "encryptionkey@0"
I20260924 00:20:09.659541 10675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "58fd7c07e2a34ced99835a57dd970ccf"
format_stamp: "Formatted at 2026-09-24 00:20:09 on dist-test-slave-777x"
server_key: "51463f6bbf322f2daa8cd2fbf0f2a287"
server_key_iv: "b1fa8da43033072447e8c91397dffe09"
server_key_version: "encryptionkey@0"
I20260924 00:20:09.661161 10675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:20:09.662032 10690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:09.662201 10675 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:20:09.662307 10675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "58fd7c07e2a34ced99835a57dd970ccf"
format_stamp: "Formatted at 2026-09-24 00:20:09 on dist-test-slave-777x"
server_key: "51463f6bbf322f2daa8cd2fbf0f2a287"
server_key_iv: "b1fa8da43033072447e8c91397dffe09"
server_key_version: "encryptionkey@0"
I20260924 00:20:09.662420 10675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:09.675429 10675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:09.677704 10675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:20:09.677953 10675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:09.678220 10675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:20:09.678547 10675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:20:09.678601 10675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:09.678646 10675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:20:09.678680 10675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:09.686118 10675 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41041
I20260924 00:20:09.686199 10803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41041 every 8 connection(s)
I20260924 00:20:09.686558 10675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 00:20:09 dist-test-slave-777x krb5kdc[10568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:20:09.695117 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 10675
I20260924 00:20:09.695256 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:20:09.695489 392 external_mini_cluster.cc:1428] Setting key 7b6c15419518050780a6f8d1dad888ad
I20260924 00:20:09.697037 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:09.687389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60847 (local address 127.0.98.62:37263)
0924 00:20:09.687464 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:09.687467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:09.687637 (+ 170us) server_negotiation.cc:407] Connection header received
0924 00:20:09.688308 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:09.688309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:09.688319 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:09.688360 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:09.689403 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:09.689936 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:09.691071 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:09.691268 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:09.691331 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:09.693746 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:20:09.693748 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:20:09.693748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:20:09.693757 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:20:09.695204 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:20:09.695737 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:20:09.695738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:20:09.695738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:20:09.695846 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:20:09.696185 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:20:09.696186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:20:09.696187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:20:09.696350 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:20:09.696409 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:09.696835 (+ 426us) server_negotiation.cc:299] Negotiation successful
0924 00:20:09.696917 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 00:20:09.697333 10806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:09.687324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37263 (local address 127.0.98.1:60847)
0924 00:20:09.687507 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:09.687549 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:09.688147 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:09.688433 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:09.688438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:09.689069 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:09.689289 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:09.689294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:09.690037 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:09.690039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:09.690982 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:09.690985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:09.691347 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:09.691348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:09.691457 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:09.691997 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:09.692007 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:09.693650 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:20:09.695374 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:20:09.695381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:20:09.695388 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:20:09.695656 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:20:09.695931 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:20:09.695932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:20:09.695932 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:20:09.696083 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:20:09.696426 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:20:09.696433 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:20:09.696668 (+ 235us) client_negotiation.cc:769] Sending connection context
0924 00:20:09.696801 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 00:20:09.697015 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":92,"threads_started":1}
I20260924 00:20:09.697908 10804 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37263
I20260924 00:20:09.698065 10804 heartbeater.cc:464] Registering TS with master...
I20260924 00:20:09.698266 10804 heartbeater.cc:511] Master 127.0.98.62:37263 requested a full tablet report, sending...
I20260924 00:20:09.698801 10613 ts_manager.cc:194] Registered new tserver with Master: 58fd7c07e2a34ced99835a57dd970ccf (127.0.98.1:41041)
I20260924 00:20:09.699375 10613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60847
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:20:09.723618 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37263
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:20:09.808705 10809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:20:09.808977 10809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:20:09.809018 10809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:20:09.810216 10809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:20:09.810297 10809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:20:09.810348 10809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:20:09.812016 10809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:20:09.812326 10809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:20:09.812605 10809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:09.817901 10815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:09.817907 10814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:09.818078 10817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:09.818126 10809 server_base.cc:1034] running on GCE node
I20260924 00:20:09.818375 10809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:20:09.818619 10809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:20:09.819772 10809 hybrid_clock.cc:648] HybridClock initialized: now 1790209209819736 us; error 55 us; skew 500 ppm
Sep 24 00:20:09 dist-test-slave-777x krb5kdc[10568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:20:09.822288 10809 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:20:09.822752 10809 webserver.cc:460] Webserver started at http://127.0.98.2:45033/ using document root <none> and password file <none>
I20260924 00:20:09.822940 10809 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:09.822979 10809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:09.823073 10809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:20:09.824079 10809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "c8a7fd8f466b487f87fbafdaf496a669"
format_stamp: "Formatted at 2026-09-24 00:20:09 on dist-test-slave-777x"
server_key: "2a588fd5862ea77d5ed51f6cc2aa4b59"
server_key_iv: "5844e3df6eeec0e4e5a0232db9cf853d"
server_key_version: "encryptionkey@0"
I20260924 00:20:09.824457 10809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c8a7fd8f466b487f87fbafdaf496a669"
format_stamp: "Formatted at 2026-09-24 00:20:09 on dist-test-slave-777x"
server_key: "2a588fd5862ea77d5ed51f6cc2aa4b59"
server_key_iv: "5844e3df6eeec0e4e5a0232db9cf853d"
server_key_version: "encryptionkey@0"
I20260924 00:20:09.825747 10809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:20:09.826488 10824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:09.826670 10809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:20:09.826771 10809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "c8a7fd8f466b487f87fbafdaf496a669"
format_stamp: "Formatted at 2026-09-24 00:20:09 on dist-test-slave-777x"
server_key: "2a588fd5862ea77d5ed51f6cc2aa4b59"
server_key_iv: "5844e3df6eeec0e4e5a0232db9cf853d"
server_key_version: "encryptionkey@0"
I20260924 00:20:09.826836 10809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:09.856314 10809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:09.858767 10809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:20:09.858906 10809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:09.859175 10809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:20:09.859522 10809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:20:09.859556 10809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:09.859583 10809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:20:09.859625 10809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:09.865407 10809 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42283
I20260924 00:20:09.865510 10937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42283 every 8 connection(s)
I20260924 00:20:09.865842 10809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:20:09.869398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 10809
I20260924 00:20:09.869546 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:20:09.869781 392 external_mini_cluster.cc:1428] Setting key 0072a5ffac048d5774ff3546e8806173
Sep 24 00:20:09 dist-test-slave-777x krb5kdc[10568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:20:09.877039 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:09.866717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33309 (local address 127.0.98.62:37263)
0924 00:20:09.866793 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:09.866795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:09.867047 (+ 252us) server_negotiation.cc:407] Connection header received
0924 00:20:09.867645 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:09.867647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:09.867659 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:09.867721 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:09.868845 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:09.869373 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:09.871103 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:09.871285 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:09.871349 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:09.873958 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:20:09.873961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:20:09.873961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:20:09.873970 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:20:09.875495 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:20:09.875939 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:20:09.875940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:20:09.875941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:20:09.876002 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:20:09.876299 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:20:09.876300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:20:09.876300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:20:09.876423 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:20:09.876501 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:09.876779 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 00:20:09.876938 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 00:20:09.877154 10940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:09.866649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37263 (local address 127.0.98.2:33309)
0924 00:20:09.866907 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:09.866954 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:09.867486 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:09.867789 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:09.867792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:09.868506 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:09.868731 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:09.868736 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:09.869476 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:09.869478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:09.870999 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:09.871003 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:09.871377 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:09.871378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:09.871549 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:09.872252 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:09.872265 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:09.873847 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:20:09.875610 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:20:09.875617 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:20:09.875619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:20:09.875850 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:20:09.876076 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:20:09.876077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:20:09.876077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:20:09.876243 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:20:09.876492 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:20:09.876498 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:20:09.876644 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 00:20:09.876745 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 00:20:09.876910 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":99,"threads_started":1}
I20260924 00:20:09.877568 10938 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37263
I20260924 00:20:09.877662 10938 heartbeater.cc:464] Registering TS with master...
I20260924 00:20:09.877861 10938 heartbeater.cc:511] Master 127.0.98.62:37263 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 00:20:09.878212 10613 ts_manager.cc:194] Registered new tserver with Master: c8a7fd8f466b487f87fbafdaf496a669 (127.0.98.2:42283)
I20260924 00:20:09.878593 10613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:20:09.895711 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37263
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:20:09.977128 10943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:20:09.977339 10943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:20:09.977363 10943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:20:09.978451 10943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:20:09.978488 10943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:20:09.978513 10943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:20:09.979859 10943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:20:09.980108 10943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:20:09.980376 10943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:09.985215 10948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:09.985225 10951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:09.985219 10949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:09.985392 10943 server_base.cc:1034] running on GCE node
I20260924 00:20:09.985586 10943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:20:09.985816 10943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:20:09.987078 10943 hybrid_clock.cc:648] HybridClock initialized: now 1790209209987037 us; error 50 us; skew 500 ppm
Sep 24 00:20:09 dist-test-slave-777x krb5kdc[10568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:20:09.989879 10943 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:20:09.990396 10943 webserver.cc:460] Webserver started at http://127.0.98.3:35247/ using document root <none> and password file <none>
I20260924 00:20:09.990638 10943 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:09.990736 10943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:09.990911 10943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:20:09.992050 10943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "0c197d42937448c1bcf846a452ec8423"
format_stamp: "Formatted at 2026-09-24 00:20:09 on dist-test-slave-777x"
server_key: "5ea81942752b1ac3822d60f19ccfd504"
server_key_iv: "e159870e4f07831cb7fa1b9d41f91b13"
server_key_version: "encryptionkey@0"
I20260924 00:20:09.992502 10943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0c197d42937448c1bcf846a452ec8423"
format_stamp: "Formatted at 2026-09-24 00:20:09 on dist-test-slave-777x"
server_key: "5ea81942752b1ac3822d60f19ccfd504"
server_key_iv: "e159870e4f07831cb7fa1b9d41f91b13"
server_key_version: "encryptionkey@0"
I20260924 00:20:09.993988 10943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:20:09.994750 10958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:09.994920 10943 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:20:09.995021 10943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "0c197d42937448c1bcf846a452ec8423"
format_stamp: "Formatted at 2026-09-24 00:20:09 on dist-test-slave-777x"
server_key: "5ea81942752b1ac3822d60f19ccfd504"
server_key_iv: "e159870e4f07831cb7fa1b9d41f91b13"
server_key_version: "encryptionkey@0"
I20260924 00:20:09.995103 10943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:10.005769 10943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:10.007925 10943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:20:10.008111 10943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:10.008392 10943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:20:10.008725 10943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:20:10.008782 10943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:10.008828 10943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:20:10.008872 10943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:10.015637 10943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40015
I20260924 00:20:10.015710 11071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40015 every 8 connection(s)
I20260924 00:20:10.016086 10943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:20:10.020676 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 10943
I20260924 00:20:10.020782 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:20:10.020946 392 external_mini_cluster.cc:1428] Setting key 748233685f0130e9a8074adbb6e5ff2e
Sep 24 00:20:10 dist-test-slave-777x krb5kdc[10568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:20:10.025076 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:10.016857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58967 (local address 127.0.98.62:37263)
0924 00:20:10.016917 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:10.016920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:10.017086 (+ 166us) server_negotiation.cc:407] Connection header received
0924 00:20:10.017578 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:10.017579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:10.017589 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:10.017643 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:10.018617 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.019208 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.020244 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.020451 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.020566 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:10.022718 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:20:10.022719 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:20:10.022719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:20:10.022725 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:20:10.023977 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:20:10.024374 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:20:10.024374 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:20:10.024375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:20:10.024421 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:20:10.024617 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:20:10.024618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:20:10.024618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:20:10.024704 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:20:10.024748 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:10.024952 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 00:20:10.025013 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 00:20:10.025246 11074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:10.016794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37263 (local address 127.0.98.3:58967)
0924 00:20:10.016950 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:10.016992 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:10.017467 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:10.017702 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:10.017705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:10.018279 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:10.018476 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:10.018480 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.019305 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.019306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:10.020105 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:10.020107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.020579 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.020579 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:10.020679 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:10.021289 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:10.021297 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:10.022630 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:20:10.024053 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:20:10.024059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:20:10.024060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:20:10.024321 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:20:10.024476 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:20:10.024477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:20:10.024477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:20:10.024575 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:20:10.024749 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:20:10.024752 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:20:10.024860 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 00:20:10.024938 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 00:20:10.025052 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260924 00:20:10.025655 11072 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37263
I20260924 00:20:10.025738 11072 heartbeater.cc:464] Registering TS with master...
I20260924 00:20:10.025925 11072 heartbeater.cc:511] Master 127.0.98.62:37263 requested a full tablet report, sending...
I20260924 00:20:10.026289 10613 ts_manager.cc:194] Registered new tserver with Master: 0c197d42937448c1bcf846a452ec8423 (127.0.98.3:40015)
I20260924 00:20:10.026665 10613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58967
I20260924 00:20:10.032838 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:20:10.040588 10652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:10.034257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33534 (local address 127.0.98.62:37263)
0924 00:20:10.034341 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:10.034344 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:10.034408 (+ 64us) server_negotiation.cc:407] Connection header received
0924 00:20:10.034466 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:10.034467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:10.034475 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:10.034528 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:10.035108 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.035581 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.036530 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.036714 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.036770 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:10.038092 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:20:10.038093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:20:10.038093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:20:10.038100 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:20:10.039410 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:20:10.039747 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:20:10.039748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:20:10.039748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:20:10.039797 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:20:10.040055 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:20:10.040056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:20:10.040056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:20:10.040246 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:20:10.040325 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:10.040430 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 00:20:10.040491 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 00:20:10.042306 10613 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:20:10.043843 10613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:20:10.050109 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:10.046090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41041 (local address 127.0.0.1:33076)
0924 00:20:10.046259 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:10.046303 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:10.046375 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:10.046721 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:10.046724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:10.046726 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:20:10.046797 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:10.046799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.047751 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.047752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:10.048930 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:10.048933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.049792 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.049793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:10.049869 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:10.049883 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 00:20:10.049977 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 00:20:10.050032 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":56,"threads_started":1}
I20260924 00:20:10.050122 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:10.046196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33076 (local address 127.0.98.1:41041)
0924 00:20:10.046476 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:10.046480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:10.046490 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:20:10.046573 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:10.046575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:10.046591 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:10.046714 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:20:10.046880 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.047641 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.049044 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.049700 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.049789 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:10.049851 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:10.049995 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 00:20:10.050040 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":72,"threads_started":1}
I20260924 00:20:10.050647 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:10.046103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42283 (local address 127.0.0.1:39090)
0924 00:20:10.046366 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:10.046386 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:10.046441 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:10.046762 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:10.046763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:10.046764 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:20:10.046798 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:10.046799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.048209 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.048210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:10.049423 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:10.049425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.050393 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.050394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:10.050459 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:10.050462 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:20:10.050510 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 00:20:10.050566 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260924 00:20:10.050743 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:10.046480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49714 (local address 127.0.98.3:40015)
0924 00:20:10.046661 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:10.046666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:10.046876 (+ 210us) server_negotiation.cc:407] Connection header received
0924 00:20:10.046921 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:10.046924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:10.046939 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:10.047012 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:20:10.047212 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.048318 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.049409 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.050123 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.050387 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:10.050441 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:10.050620 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 00:20:10.050663 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260924 00:20:10.050971 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:10.046204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39090 (local address 127.0.98.2:42283)
0924 00:20:10.046400 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:10.046406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:10.046495 (+ 89us) server_negotiation.cc:407] Connection header received
0924 00:20:10.046583 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:10.046586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:10.046604 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:10.046680 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:20:10.046972 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.048086 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.049488 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.050208 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.050302 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:10.050361 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:10.050847 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 00:20:10.050887 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":82,"threads_started":1}
I20260924 00:20:10.050899 10738 tablet_service.cc:1467] Processing CreateTablet for tablet e3d4040f86c4469aad65c6c15e32a8d7 (DEFAULT_TABLE table=test-table [id=5b47f655bfdc440091bca7b6b0d1e5e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:20:10.051249 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:10.046287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40015 (local address 127.0.0.1:49714)
0924 00:20:10.046649 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:10.046660 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:10.046712 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:10.047098 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:10.047099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:10.047101 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:20:10.047135 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:10.047137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.048411 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.048412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:10.049308 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:10.049310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.050464 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.050465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:10.050539 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:10.050542 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:20:10.050594 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 00:20:10.050627 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":67,"threads_started":1}
I20260924 00:20:10.051266 10738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d4040f86c4469aad65c6c15e32a8d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:10.051571 10872 tablet_service.cc:1467] Processing CreateTablet for tablet e3d4040f86c4469aad65c6c15e32a8d7 (DEFAULT_TABLE table=test-table [id=5b47f655bfdc440091bca7b6b0d1e5e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:20:10.051810 10872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d4040f86c4469aad65c6c15e32a8d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:10.051918 11006 tablet_service.cc:1467] Processing CreateTablet for tablet e3d4040f86c4469aad65c6c15e32a8d7 (DEFAULT_TABLE table=test-table [id=5b47f655bfdc440091bca7b6b0d1e5e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:20:10.052239 11006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3d4040f86c4469aad65c6c15e32a8d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:10.054507 11089 tablet_bootstrap.cc:492] T e3d4040f86c4469aad65c6c15e32a8d7 P c8a7fd8f466b487f87fbafdaf496a669: Bootstrap starting.
I20260924 00:20:10.054919 11090 tablet_bootstrap.cc:492] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf: Bootstrap starting.
I20260924 00:20:10.055310 11089 tablet_bootstrap.cc:654] T e3d4040f86c4469aad65c6c15e32a8d7 P c8a7fd8f466b487f87fbafdaf496a669: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:10.055456 11091 tablet_bootstrap.cc:492] T e3d4040f86c4469aad65c6c15e32a8d7 P 0c197d42937448c1bcf846a452ec8423: Bootstrap starting.
I20260924 00:20:10.055603 11090 tablet_bootstrap.cc:654] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:10.056020 11091 tablet_bootstrap.cc:654] T e3d4040f86c4469aad65c6c15e32a8d7 P 0c197d42937448c1bcf846a452ec8423: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:10.056066 11089 log.cc:824] T e3d4040f86c4469aad65c6c15e32a8d7 P c8a7fd8f466b487f87fbafdaf496a669: Log is configured to *not* fsync() on all Append() calls
I20260924 00:20:10.056227 11090 log.cc:824] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf: Log is configured to *not* fsync() on all Append() calls
I20260924 00:20:10.056646 11091 log.cc:824] T e3d4040f86c4469aad65c6c15e32a8d7 P 0c197d42937448c1bcf846a452ec8423: Log is configured to *not* fsync() on all Append() calls
I20260924 00:20:10.056844 11089 tablet_bootstrap.cc:492] T e3d4040f86c4469aad65c6c15e32a8d7 P c8a7fd8f466b487f87fbafdaf496a669: No bootstrap required, opened a new log
I20260924 00:20:10.056919 11089 ts_tablet_manager.cc:1397] T e3d4040f86c4469aad65c6c15e32a8d7 P c8a7fd8f466b487f87fbafdaf496a669: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:20:10.056996 11090 tablet_bootstrap.cc:492] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf: No bootstrap required, opened a new log
I20260924 00:20:10.057075 11090 ts_tablet_manager.cc:1397] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:20:10.057344 11091 tablet_bootstrap.cc:492] T e3d4040f86c4469aad65c6c15e32a8d7 P 0c197d42937448c1bcf846a452ec8423: No bootstrap required, opened a new log
I20260924 00:20:10.057411 11091 ts_tablet_manager.cc:1397] T e3d4040f86c4469aad65c6c15e32a8d7 P 0c197d42937448c1bcf846a452ec8423: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:20:10.058838 11091 raft_consensus.cc:348] T e3d4040f86c4469aad65c6c15e32a8d7 P 0c197d42937448c1bcf846a452ec8423 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58fd7c07e2a34ced99835a57dd970ccf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41041 } } peers { permanent_uuid: "c8a7fd8f466b487f87fbafdaf496a669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42283 } } peers { permanent_uuid: "0c197d42937448c1bcf846a452ec8423" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40015 } }
I20260924 00:20:10.059000 11091 raft_consensus.cc:374] T e3d4040f86c4469aad65c6c15e32a8d7 P 0c197d42937448c1bcf846a452ec8423 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:10.059052 11091 raft_consensus.cc:729] T e3d4040f86c4469aad65c6c15e32a8d7 P 0c197d42937448c1bcf846a452ec8423 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c197d42937448c1bcf846a452ec8423, State: Initialized, Role: FOLLOWER
I20260924 00:20:10.058962 11090 raft_consensus.cc:348] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58fd7c07e2a34ced99835a57dd970ccf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41041 } } peers { permanent_uuid: "c8a7fd8f466b487f87fbafdaf496a669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42283 } } peers { permanent_uuid: "0c197d42937448c1bcf846a452ec8423" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40015 } }
I20260924 00:20:10.058962 11089 raft_consensus.cc:348] T e3d4040f86c4469aad65c6c15e32a8d7 P c8a7fd8f466b487f87fbafdaf496a669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58fd7c07e2a34ced99835a57dd970ccf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41041 } } peers { permanent_uuid: "c8a7fd8f466b487f87fbafdaf496a669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42283 } } peers { permanent_uuid: "0c197d42937448c1bcf846a452ec8423" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40015 } }
I20260924 00:20:10.059144 11089 raft_consensus.cc:374] T e3d4040f86c4469aad65c6c15e32a8d7 P c8a7fd8f466b487f87fbafdaf496a669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:10.059144 11090 raft_consensus.cc:374] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:10.059190 11090 raft_consensus.cc:729] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58fd7c07e2a34ced99835a57dd970ccf, State: Initialized, Role: FOLLOWER
I20260924 00:20:10.059190 11089 raft_consensus.cc:729] T e3d4040f86c4469aad65c6c15e32a8d7 P c8a7fd8f466b487f87fbafdaf496a669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a7fd8f466b487f87fbafdaf496a669, State: Initialized, Role: FOLLOWER
I20260924 00:20:10.059165 11091 consensus_queue.cc:260] T e3d4040f86c4469aad65c6c15e32a8d7 P 0c197d42937448c1bcf846a452ec8423 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58fd7c07e2a34ced99835a57dd970ccf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41041 } } peers { permanent_uuid: "c8a7fd8f466b487f87fbafdaf496a669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42283 } } peers { permanent_uuid: "0c197d42937448c1bcf846a452ec8423" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40015 } }
I20260924 00:20:10.059302 11089 consensus_queue.cc:260] T e3d4040f86c4469aad65c6c15e32a8d7 P c8a7fd8f466b487f87fbafdaf496a669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58fd7c07e2a34ced99835a57dd970ccf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41041 } } peers { permanent_uuid: "c8a7fd8f466b487f87fbafdaf496a669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42283 } } peers { permanent_uuid: "0c197d42937448c1bcf846a452ec8423" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40015 } }
I20260924 00:20:10.059320 11090 consensus_queue.cc:260] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58fd7c07e2a34ced99835a57dd970ccf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41041 } } peers { permanent_uuid: "c8a7fd8f466b487f87fbafdaf496a669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42283 } } peers { permanent_uuid: "0c197d42937448c1bcf846a452ec8423" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40015 } }
I20260924 00:20:10.059439 11091 ts_tablet_manager.cc:1428] T e3d4040f86c4469aad65c6c15e32a8d7 P 0c197d42937448c1bcf846a452ec8423: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:20:10.059581 11089 ts_tablet_manager.cc:1428] T e3d4040f86c4469aad65c6c15e32a8d7 P c8a7fd8f466b487f87fbafdaf496a669: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:20:10.059572 11090 ts_tablet_manager.cc:1428] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 00:20:10.059698 10804 heartbeater.cc:503] Master 127.0.98.62:37263 was elected leader, sending a full tablet report...
I20260924 00:20:10.059725 11072 heartbeater.cc:503] Master 127.0.98.62:37263 was elected leader, sending a full tablet report...
I20260924 00:20:10.059993 10938 heartbeater.cc:503] Master 127.0.98.62:37263 was elected leader, sending a full tablet report...
W20260924 00:20:10.116395 10939 tablet.cc:2369] T e3d4040f86c4469aad65c6c15e32a8d7 P c8a7fd8f466b487f87fbafdaf496a669: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:20:10.187346 10805 tablet.cc:2369] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:20:10.266801 11073 tablet.cc:2369] T e3d4040f86c4469aad65c6c15e32a8d7 P 0c197d42937448c1bcf846a452ec8423: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:20:10.301077 11097 raft_consensus.cc:482] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:10.301214 11097 raft_consensus.cc:504] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58fd7c07e2a34ced99835a57dd970ccf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41041 } } peers { permanent_uuid: "c8a7fd8f466b487f87fbafdaf496a669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42283 } } peers { permanent_uuid: "0c197d42937448c1bcf846a452ec8423" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40015 } }
I20260924 00:20:10.301564 11097 leader_election.cc:288] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8a7fd8f466b487f87fbafdaf496a669 (127.0.98.2:42283), 0c197d42937448c1bcf846a452ec8423 (127.0.98.3:40015)
I20260924 00:20:10.304679 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:10.301776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42283 (local address 127.0.98.1:56101)
0924 00:20:10.301942 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:10.301955 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:10.302016 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:10.302171 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:10.302172 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:10.302175 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:20:10.302258 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:10.302260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.302909 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.302911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:10.303797 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:10.303799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.304420 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.304421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:10.304501 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:10.304506 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:20:10.304570 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 00:20:10.304605 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":15,"thread_start_us":66,"threads_started":1}
I20260924 00:20:10.304741 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:10.301942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56101 (local address 127.0.98.2:42283)
0924 00:20:10.301992 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:10.301994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:10.302007 (+ 13us) server_negotiation.cc:407] Connection header received
0924 00:20:10.302086 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:10.302087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:10.302097 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:10.302141 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:20:10.302334 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.302817 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.303938 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.304336 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.304396 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:10.304405 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:10.304594 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 00:20:10.304646 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 00:20:10.304857 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:10.301776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40015 (local address 127.0.98.1:46273)
0924 00:20:10.301911 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:10.301923 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:10.301986 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:10.302207 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:10.302208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:10.302210 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:20:10.302283 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:10.302284 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.303042 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.303043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:10.303804 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:10.303806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.304531 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.304532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:10.304637 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:10.304642 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:20:10.304717 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 00:20:10.304762 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
I20260924 00:20:10.304878 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:10.301999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46273 (local address 127.0.98.3:40015)
0924 00:20:10.302044 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:10.302047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:10.302057 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:20:10.302086 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:10.302087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:10.302099 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:10.302188 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:20:10.302377 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.302940 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.303959 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.304424 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.304501 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:10.304511 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:10.304735 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 00:20:10.304781 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:20:10.305123 10892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d4040f86c4469aad65c6c15e32a8d7" candidate_uuid: "58fd7c07e2a34ced99835a57dd970ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a7fd8f466b487f87fbafdaf496a669" is_pre_election: true
I20260924 00:20:10.305182 11026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d4040f86c4469aad65c6c15e32a8d7" candidate_uuid: "58fd7c07e2a34ced99835a57dd970ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c197d42937448c1bcf846a452ec8423" is_pre_election: true
I20260924 00:20:10.305299 10892 raft_consensus.cc:2454] T e3d4040f86c4469aad65c6c15e32a8d7 P c8a7fd8f466b487f87fbafdaf496a669 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58fd7c07e2a34ced99835a57dd970ccf in term 0.
I20260924 00:20:10.305368 11026 raft_consensus.cc:2454] T e3d4040f86c4469aad65c6c15e32a8d7 P 0c197d42937448c1bcf846a452ec8423 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58fd7c07e2a34ced99835a57dd970ccf in term 0.
I20260924 00:20:10.305586 10691 leader_election.cc:302] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58fd7c07e2a34ced99835a57dd970ccf, c8a7fd8f466b487f87fbafdaf496a669; no voters:
I20260924 00:20:10.305737 11097 raft_consensus.cc:2781] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:10.305796 11097 raft_consensus.cc:482] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:20:10.305830 11097 raft_consensus.cc:3037] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:10.306610 11097 raft_consensus.cc:504] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58fd7c07e2a34ced99835a57dd970ccf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41041 } } peers { permanent_uuid: "c8a7fd8f466b487f87fbafdaf496a669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42283 } } peers { permanent_uuid: "0c197d42937448c1bcf846a452ec8423" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40015 } }
I20260924 00:20:10.306766 11097 leader_election.cc:288] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf [CANDIDATE]: Term 1 election: Requested vote from peers c8a7fd8f466b487f87fbafdaf496a669 (127.0.98.2:42283), 0c197d42937448c1bcf846a452ec8423 (127.0.98.3:40015)
I20260924 00:20:10.306974 11026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d4040f86c4469aad65c6c15e32a8d7" candidate_uuid: "58fd7c07e2a34ced99835a57dd970ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c197d42937448c1bcf846a452ec8423"
I20260924 00:20:10.306952 10892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3d4040f86c4469aad65c6c15e32a8d7" candidate_uuid: "58fd7c07e2a34ced99835a57dd970ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a7fd8f466b487f87fbafdaf496a669"
I20260924 00:20:10.307054 11026 raft_consensus.cc:3037] T e3d4040f86c4469aad65c6c15e32a8d7 P 0c197d42937448c1bcf846a452ec8423 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:10.307054 10892 raft_consensus.cc:3037] T e3d4040f86c4469aad65c6c15e32a8d7 P c8a7fd8f466b487f87fbafdaf496a669 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:10.308043 11026 raft_consensus.cc:2454] T e3d4040f86c4469aad65c6c15e32a8d7 P 0c197d42937448c1bcf846a452ec8423 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58fd7c07e2a34ced99835a57dd970ccf in term 1.
I20260924 00:20:10.308043 10892 raft_consensus.cc:2454] T e3d4040f86c4469aad65c6c15e32a8d7 P c8a7fd8f466b487f87fbafdaf496a669 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58fd7c07e2a34ced99835a57dd970ccf in term 1.
I20260924 00:20:10.308279 10692 leader_election.cc:302] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c197d42937448c1bcf846a452ec8423, 58fd7c07e2a34ced99835a57dd970ccf; no voters:
I20260924 00:20:10.308439 11097 raft_consensus.cc:2781] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:10.308594 11097 raft_consensus.cc:686] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf [term 1 LEADER]: Becoming Leader. State: Replica: 58fd7c07e2a34ced99835a57dd970ccf, State: Running, Role: LEADER
I20260924 00:20:10.308698 11097 consensus_queue.cc:237] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58fd7c07e2a34ced99835a57dd970ccf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41041 } } peers { permanent_uuid: "c8a7fd8f466b487f87fbafdaf496a669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42283 } } peers { permanent_uuid: "0c197d42937448c1bcf846a452ec8423" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40015 } }
I20260924 00:20:10.309633 10612 catalog_manager.cc:5495] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf reported cstate change: term changed from 0 to 1, leader changed from <none> to 58fd7c07e2a34ced99835a57dd970ccf (127.0.98.1). New cstate: current_term: 1 leader_uuid: "58fd7c07e2a34ced99835a57dd970ccf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58fd7c07e2a34ced99835a57dd970ccf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a7fd8f466b487f87fbafdaf496a669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c197d42937448c1bcf846a452ec8423" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40015 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:10.332840 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:10.330384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33090 (local address 127.0.98.1:41041)
0924 00:20:10.330449 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:10.330451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:10.330462 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:20:10.330528 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:10.330529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:10.330549 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:10.330608 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:20:10.330736 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.331231 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.332028 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:10.332272 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:10.332345 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:10.332505 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:20:10.332615 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:20:10.332709 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:10.332734 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 00:20:10.332754 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 00:20:10.334760 10892 raft_consensus.cc:1261] T e3d4040f86c4469aad65c6c15e32a8d7 P c8a7fd8f466b487f87fbafdaf496a669 [term 1 FOLLOWER]: Refusing update from remote peer 58fd7c07e2a34ced99835a57dd970ccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:20:10.334794 11026 raft_consensus.cc:1261] T e3d4040f86c4469aad65c6c15e32a8d7 P 0c197d42937448c1bcf846a452ec8423 [term 1 FOLLOWER]: Refusing update from remote peer 58fd7c07e2a34ced99835a57dd970ccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:20:10.335120 11100 consensus_queue.cc:1035] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c197d42937448c1bcf846a452ec8423" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:10.335258 11097 consensus_queue.cc:1035] T e3d4040f86c4469aad65c6c15e32a8d7 P 58fd7c07e2a34ced99835a57dd970ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a7fd8f466b487f87fbafdaf496a669" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:10.338615 11103 mvcc.cc:205] Tried to move back new op lower bound from 7332696925528215552 to 7332696925425229824. Current Snapshot: MvccSnapshot[applied={T|T < 7332696925528215552 or (T in {7332696925528215552})}]
I20260924 00:20:10.338734 11104 mvcc.cc:205] Tried to move back new op lower bound from 7332696925528215552 to 7332696925425229824. Current Snapshot: MvccSnapshot[applied={T|T < 7332696925528215552 or (T in {7332696925528215552})}]
I20260924 00:20:10.339986 10612 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:20:10.340093 10612 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:20:10.340915 10612 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4102f9cbb274166ac3676fee97673b3: Sending DeleteTablet for 3 replicas of tablet e3d4040f86c4469aad65c6c15e32a8d7
I20260924 00:20:10.341288 10872 tablet_service.cc:1514] Processing DeleteTablet for tablet e3d4040f86c4469aad65c6c15e32a8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:20:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39090
I20260924 00:20:10.341317 10738 tablet_service.cc:1514] Processing DeleteTablet for tablet e3d4040f86c4469aad65c6c15e32a8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:20:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33076
I20260924 00:20:10.341517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 10675
I20260924 00:20:10.341562 11006 tablet_service.cc:1514] Processing DeleteTablet for tablet e3d4040f86c4469aad65c6c15e32a8d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:20:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49714
I20260924 00:20:10.344002 11115 tablet_replica.cc:330] T e3d4040f86c4469aad65c6c15e32a8d7 P 0c197d42937448c1bcf846a452ec8423: stopping tablet replica
I20260924 00:20:10.344136 11115 raft_consensus.cc:2229] T e3d4040f86c4469aad65c6c15e32a8d7 P 0c197d42937448c1bcf846a452ec8423 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:20:10.344280 11115 raft_consensus.cc:2258] T e3d4040f86c4469aad65c6c15e32a8d7 P 0c197d42937448c1bcf846a452ec8423 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:10.344735 11115 ts_tablet_manager.cc:1905] T e3d4040f86c4469aad65c6c15e32a8d7 P 0c197d42937448c1bcf846a452ec8423: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:20:10.346834 11115 ts_tablet_manager.cc:1918] T e3d4040f86c4469aad65c6c15e32a8d7 P 0c197d42937448c1bcf846a452ec8423: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:20:10.346916 11115 log.cc:1196] T e3d4040f86c4469aad65c6c15e32a8d7 P 0c197d42937448c1bcf846a452ec8423: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/e3d4040f86c4469aad65c6c15e32a8d7
I20260924 00:20:10.347203 11115 ts_tablet_manager.cc:1939] T e3d4040f86c4469aad65c6c15e32a8d7 P 0c197d42937448c1bcf846a452ec8423: Deleting consensus metadata
I20260924 00:20:10.347704 10598 catalog_manager.cc:4841] TS 0c197d42937448c1bcf846a452ec8423 (127.0.98.3:40015): tablet e3d4040f86c4469aad65c6c15e32a8d7 (table test-table [id=5b47f655bfdc440091bca7b6b0d1e5e1]) successfully deleted
W20260924 00:20:10.348269 10598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41041 (error 108)
I20260924 00:20:10.348793 11113 tablet_replica.cc:330] T e3d4040f86c4469aad65c6c15e32a8d7 P c8a7fd8f466b487f87fbafdaf496a669: stopping tablet replica
I20260924 00:20:10.348826 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:10.348683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41041 (local address 127.0.0.1:33104)
0924 00:20:10.348759 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:10.348798 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 00:20:10.348898 11113 raft_consensus.cc:2229] T e3d4040f86c4469aad65c6c15e32a8d7 P c8a7fd8f466b487f87fbafdaf496a669 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 00:20:10.348985 10598 catalog_manager.cc:4584] TS 58fd7c07e2a34ced99835a57dd970ccf (127.0.98.1:41041): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e3d4040f86c4469aad65c6c15e32a8d7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41041: connect: Connection refused (error 111)
I20260924 00:20:10.349010 11113 raft_consensus.cc:2258] T e3d4040f86c4469aad65c6c15e32a8d7 P c8a7fd8f466b487f87fbafdaf496a669 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:10.349213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 10809
I20260924 00:20:10.353794 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 10943
I20260924 00:20:10.353832 11083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:10.353700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42283 (local address 127.0.0.1:39100)
0924 00:20:10.353757 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:10.353792 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 00:20:10.358829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 10581
2026-09-24T00:20:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2213 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:20:10 dist-test-slave-777x krb5kdc[11119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:20:10 dist-test-slave-777x krb5kdc[11119](info): set up 2 sockets
Sep 24 00:20:10 dist-test-slave-777x krb5kdc[11119](info): commencing operation
krb5kdc: starting...
W20260924 00:20:11.473344 392 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:20:11 dist-test-slave-777x krb5kdc[11119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:20:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:20:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 00:20:11.647289 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskYEWBIr/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/postgres -l logfile start
2026-09-24 00:20:14.457 UTC [11149] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 00:20:14.457 UTC [11149] LOG: listening on IPv4 address "127.0.98.20", port 51887
2026-09-24 00:20:14.462 UTC [11149] LOG: listening on Unix socket "/tmp/.s.PGSQL.51887"
2026-09-24 00:20:14.485 UTC [11152] LOG: database system was shut down at 2026-09-24 00:20:12 UTC
2026-09-24 00:20:14.489 UTC [11149] LOG: database system is ready to accept connections
I20260924 00:20:15.521798 392 mini_postgres.cc:96] Postgres bound to 51887
2026-09-24 00:20:15.528 UTC [11160] FATAL: database "slave" does not exist
127.0.98.20:51887 - accepting connections
I20260924 00:20:15.528698 392 mini_ranger.cc:161] Starting Ranger...
I20260924 00:20:15.545027 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 00:20:16.141752 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 00:20:16.141841 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ranger-admin
2026-09-24 00:20:16,452 [I] DB FLAVOR :POSTGRES
2026-09-24 00:20:16,452 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 00:20:16,452 [I] Checking connection..
2026-09-24 00:20:16,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 00:20:16,713 [I] Checking connection passed.
2026-09-24 00:20:16,714 [I] --------- Verifying version history table ---------
2026-09-24 00:20:16,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 00:20:16,948 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 00:20:16,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 00:20:17,193 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 00:20:17,193 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 00:20:17,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 00:20:17.443 UTC [11224] WARNING: there is no transaction in progress
2026-09-24 00:20:17,450 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 00:20:17,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 00:20:17,697 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 00:20:17,698 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 00:20:17,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 00:20:17,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 00:20:18,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-777x.c.gcp-upstream.internal','N') ;"
2026-09-24 00:20:18,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 00:20:18,742 [I] Table x_portal_user does not exist in database ranger
2026-09-24 00:20:18,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 00:20:19,001 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 00:20:19,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 00:20:19,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 00:20:19,511 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 00:20:19,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 00:20:20.589 UTC [11332] WARNING: there is no transaction in progress
2026-09-24 00:20:20.607 UTC [11332] WARNING: there is no transaction in progress
2026-09-24 00:20:20.626 UTC [11332] WARNING: there is no transaction in progress
2026-09-24 00:20:20.645 UTC [11332] WARNING: there is no transaction in progress
2026-09-24 00:20:20.662 UTC [11332] WARNING: there is no transaction in progress
2026-09-24 00:20:20.683 UTC [11332] WARNING: there is no transaction in progress
2026-09-24 00:20:20.750 UTC [11332] WARNING: there is no transaction in progress
2026-09-24 00:20:20.757 UTC [11332] WARNING: there is no transaction in progress
2026-09-24 00:20:20.775 UTC [11332] WARNING: there is no transaction in progress
2026-09-24 00:20:20.786 UTC [11332] WARNING: there is no transaction in progress
2026-09-24 00:20:20.798 UTC [11332] WARNING: there is no transaction in progress
2026-09-24 00:20:20.813 UTC [11332] WARNING: there is no transaction in progress
2026-09-24 00:20:20.823 UTC [11332] WARNING: there is no transaction in progress
2026-09-24 00:20:20.830 UTC [11332] WARNING: there is no transaction in progress
2026-09-24 00:20:20.838 UTC [11332] WARNING: there is no transaction in progress
2026-09-24 00:20:20.846 UTC [11332] WARNING: there is no transaction in progress
2026-09-24 00:20:21,591 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 00:20:21,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 00:20:21,842 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 00:20:21,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 00:20:22,096 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 00:20:22,096 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 00:20:22,346 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 00:20:22,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 00:20:22,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 00:20:22,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-777x.c.gcp-upstream.internal';"
2026-09-24 00:20:23,124 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 00:20:23,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 00:20:23,377 [I] DB_PATCHES have already been applied
I20260924 00:20:23.384080 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskYEWBIr/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 00:20:23.384289 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 00:20:23.387539 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:54479
Sep 24, 2026 12:20:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 12:20:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 12:20:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 12:20:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 12:20:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 12:20:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 12:20:24 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 00:20:24.652652 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.124s user 0.000s sys 0.009s
Sep 24, 2026 12:20:45 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 12:20:45 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 12:20:45 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 12:20:45 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 12:20:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 12:20:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 12:20:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 12:20:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 12:20:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 12:20:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 12:20:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 12:20:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 12:20:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 12:20:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 12:20:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 12:20:45 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 12:20:45 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 12:20:46 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 00:20:47.521886 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 00:20:47 dist-test-slave-777x krb5kdc[11119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209247, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 00:20:47.614777 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 00:20:47.625257 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 00:20:48.729607 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 00:20:48.729699 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ranger-kms
I20260924 00:20:48.729722 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:51887
2026-09-24 00:20:48,912 [I] DB FLAVOR :POSTGRES
2026-09-24 00:20:48,912 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 00:20:48,912 [I] Checking connection
2026-09-24 00:20:48,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 00:20:49,189 [I] connection success
2026-09-24 00:20:49,189 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 00:20:49,189 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 00:20:49,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 00:20:49,460 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 00:20:49,460 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 00:20:49,461 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 00:20:49,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51887/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 00:20:49,747 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 00:20:50.309053 392 mini_ranger_kms.cc:310] Created kms service
I20260924 00:20:50.518354 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 00:20:50.570566 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 00:20:50 dist-test-slave-777x krb5kdc[11119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209247, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 00:20:51.009807 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 00:20:51.009935 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 00:20:51.009960 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 00:20:51.013373 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11513
I20260924 00:20:51.013513 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:45395
Sep 24, 2026 12:20:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 12:20:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 12:20:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 12:20:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 12:20:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 12:20:52 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 00:20:53.325356 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.711s user 0.008s sys 0.154s
I20260924 00:20:53.325496 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 00:20:53.325583 392 mini_ranger_kms.cc:401] 127.0.98.20:45395/kms/v1/keys
Sep 24 00:20:53 dist-test-slave-777x krb5kdc[11119](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790209253, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 00:20:53 dist-test-slave-777x krb5kdc[11119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209253, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 00:20:54 dist-test-slave-777x krb5kdc[11119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209253, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 00:20:56 dist-test-slave-777x krb5kdc[11119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:20:56.605252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33377
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33377
--ranger_config_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45395/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:20:56.689536 11563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:20:56.689821 11563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:20:56.689875 11563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:20:56.691331 11563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:20:56.691391 11563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:20:56.691406 11563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:20:56.691421 11563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:20:56.691434 11563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:20:56.693213 11563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:45395/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33377
--ranger_config_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:20:56.693500 11563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:20:56.693751 11563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:56.699752 11569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:56.699741 11568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:56.699846 11563 server_base.cc:1034] running on GCE node
W20260924 00:20:56.699878 11571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:56.700376 11563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:20:56.700759 11563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:20:56.701956 11563 hybrid_clock.cc:648] HybridClock initialized: now 1790209256701934 us; error 40 us; skew 500 ppm
Sep 24 00:20:56 dist-test-slave-777x krb5kdc[11119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:20:56.705078 11563 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:20:56.705626 11563 webserver.cc:460] Webserver started at http://127.0.98.62:41639/ using document root <none> and password file <none>
I20260924 00:20:56.705873 11563 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:56.705938 11563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:56.706102 11563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 00:20:56 dist-test-slave-777x krb5kdc[11119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 00:20:56.770581 11563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "425fb57711784a3ea0ea9f0defa62ce0"
format_stamp: "Formatted at 2026-09-24 00:20:56 on dist-test-slave-777x"
server_key: "5df9aa9c032bf7572ee3caab21a43377"
server_key_iv: "1237168ab57962f89f2f25dcdfd92737"
server_key_version: "kuduclusterkey@0"
I20260924 00:20:56.771029 11563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "425fb57711784a3ea0ea9f0defa62ce0"
format_stamp: "Formatted at 2026-09-24 00:20:56 on dist-test-slave-777x"
server_key: "5df9aa9c032bf7572ee3caab21a43377"
server_key_iv: "1237168ab57962f89f2f25dcdfd92737"
server_key_version: "kuduclusterkey@0"
I20260924 00:20:56.772701 11563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260924 00:20:56.788516 11578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:56.789175 11563 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:20:56.789305 11563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "425fb57711784a3ea0ea9f0defa62ce0"
format_stamp: "Formatted at 2026-09-24 00:20:56 on dist-test-slave-777x"
server_key: "5df9aa9c032bf7572ee3caab21a43377"
server_key_iv: "1237168ab57962f89f2f25dcdfd92737"
server_key_version: "kuduclusterkey@0"
I20260924 00:20:56.789418 11563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:56.809198 11563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:56.811430 11563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:20:56.811599 11563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:56.817222 11563 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33377
I20260924 00:20:56.817373 11630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33377 every 8 connection(s)
I20260924 00:20:56.817672 11563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:20:56.818441 11631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:56.820910 11631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0: Bootstrap starting.
I20260924 00:20:56.821523 11631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:56.821853 11631 log.cc:824] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0: Log is configured to *not* fsync() on all Append() calls
I20260924 00:20:56.822008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 11563
I20260924 00:20:56.822117 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:20:56.822320 392 external_mini_cluster.cc:1428] Setting key 77d380b62901dd7d04c9e0810b8e195d
I20260924 00:20:56.822726 11631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0: No bootstrap required, opened a new log
I20260924 00:20:56.826227 11631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425fb57711784a3ea0ea9f0defa62ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33377 } }
I20260924 00:20:56.826414 11631 raft_consensus.cc:374] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:56.826449 11631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 425fb57711784a3ea0ea9f0defa62ce0, State: Initialized, Role: FOLLOWER
I20260924 00:20:56.826637 11631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425fb57711784a3ea0ea9f0defa62ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33377 } }
I20260924 00:20:56.826818 11631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:20:56.826869 11631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:20:56.826911 11631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:56.827723 11631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425fb57711784a3ea0ea9f0defa62ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33377 } }
I20260924 00:20:56.827831 11631 leader_election.cc:302] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 425fb57711784a3ea0ea9f0defa62ce0; no voters:
I20260924 00:20:56.828044 11631 leader_election.cc:288] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:20:56.828096 11636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:56.828351 11636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0 [term 1 LEADER]: Becoming Leader. State: Replica: 425fb57711784a3ea0ea9f0defa62ce0, State: Running, Role: LEADER
I20260924 00:20:56.828419 11631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:20:56.828507 11636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425fb57711784a3ea0ea9f0defa62ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33377 } }
I20260924 00:20:56.828930 11636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "425fb57711784a3ea0ea9f0defa62ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425fb57711784a3ea0ea9f0defa62ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33377 } } }
I20260924 00:20:56.829087 11636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0 [sys.catalog]: This master's current role is: LEADER
I20260924 00:20:56.829286 11636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 425fb57711784a3ea0ea9f0defa62ce0. Latest consensus state: current_term: 1 leader_uuid: "425fb57711784a3ea0ea9f0defa62ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425fb57711784a3ea0ea9f0defa62ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33377 } } }
I20260924 00:20:56.829420 11636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0 [sys.catalog]: This master's current role is: LEADER
I20260924 00:20:56.829603 11645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:20:56.829890 11631 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 00:20:56.830209 11645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 00:20:56 dist-test-slave-777x krb5kdc[11119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:20:56.838325 11654 mvcc.cc:205] Tried to move back new op lower bound from 7332697115994984448 to 7332697115970228224. Current Snapshot: MvccSnapshot[applied={T|T < 7332697115994984448}]
I20260924 00:20:56.839335 11645 catalog_manager.cc:1347] Generated new cluster ID: 6e0579e74d674a47858f02c6d5d2e4ca
I20260924 00:20:56.839445 11645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:20:56.840030 11635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:56.822880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43308 (local address 127.0.98.62:33377)
0924 00:20:56.823056 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:56.823077 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:56.823166 (+ 89us) server_negotiation.cc:407] Connection header received
0924 00:20:56.824648 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:56.824657 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:56.824717 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:56.824899 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:56.825582 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:56.826644 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:56.827609 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:56.828239 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:56.830423 (+ 2184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:56.833046 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:20:56.833114 (+ 68us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:20:56.833114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:20:56.833222 (+ 108us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:20:56.836991 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:20:56.837556 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:20:56.837560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:20:56.837561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:20:56.837687 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:20:56.837991 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:20:56.837991 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:20:56.837992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:20:56.838461 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:20:56.839231 (+ 770us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:56.839318 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 00:20:56.839737 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":95,"threads_started":1}
I20260924 00:20:56.862793 11645 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:20:56.863749 11645 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:20:56.869076 11645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0: Generated new TSK 0
I20260924 00:20:56.869405 11645 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:20:57.640096 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45395/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33377
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:20:57.748693 11676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:20:57.748940 11676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:20:57.748981 11676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:20:57.750279 11676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:20:57.750352 11676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:20:57.750401 11676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:20:57.751940 11676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:45395/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:20:57.752231 11676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:20:57.752488 11676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:57.758206 11682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:57.758369 11676 server_base.cc:1034] running on GCE node
W20260924 00:20:57.758206 11684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:57.758255 11681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:57.758903 11676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:20:57.759195 11676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:20:57.762468 11676 hybrid_clock.cc:648] HybridClock initialized: now 1790209257762442 us; error 37 us; skew 500 ppm
Sep 24 00:20:57 dist-test-slave-777x krb5kdc[11119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:20:57.765476 11676 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:20:57.765977 11676 webserver.cc:460] Webserver started at http://127.0.98.1:42357/ using document root <none> and password file <none>
I20260924 00:20:57.766237 11676 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:57.766309 11676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:57.766460 11676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 00:20:57 dist-test-slave-777x krb5kdc[11119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 00:20:57.780035 11676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "7c08b1d2a7cf4461b1b5a2644b372f13"
format_stamp: "Formatted at 2026-09-24 00:20:57 on dist-test-slave-777x"
server_key: "8d2b7ffd2eabf81aef968671d470edd6"
server_key_iv: "7d07b886cfe857f12248f021bf581939"
server_key_version: "kuduclusterkey@0"
I20260924 00:20:57.780668 11676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7c08b1d2a7cf4461b1b5a2644b372f13"
format_stamp: "Formatted at 2026-09-24 00:20:57 on dist-test-slave-777x"
server_key: "8d2b7ffd2eabf81aef968671d470edd6"
server_key_iv: "7d07b886cfe857f12248f021bf581939"
server_key_version: "kuduclusterkey@0"
I20260924 00:20:57.783139 11676 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 00:20:57.796867 11692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:57.797582 11676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:20:57.797720 11676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "7c08b1d2a7cf4461b1b5a2644b372f13"
format_stamp: "Formatted at 2026-09-24 00:20:57 on dist-test-slave-777x"
server_key: "8d2b7ffd2eabf81aef968671d470edd6"
server_key_iv: "7d07b886cfe857f12248f021bf581939"
server_key_version: "kuduclusterkey@0"
I20260924 00:20:57.797833 11676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:57.820474 11676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:57.823046 11676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:20:57.823280 11676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:57.823619 11676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:20:57.824040 11676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:20:57.824131 11676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:57.824265 11676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:20:57.824350 11676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:57.831358 11676 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39137
I20260924 00:20:57.831894 11676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:20:57.834395 11806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39137 every 8 connection(s)
I20260924 00:20:57.838791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 11676
I20260924 00:20:57.838904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:20:57.839114 392 external_mini_cluster.cc:1428] Setting key a70155d70481d230c5bcac5bfe5ac7fc
Sep 24 00:20:57 dist-test-slave-777x krb5kdc[11119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:20:57.848910 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:57.835039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33377 (local address 127.0.98.1:37187)
0924 00:20:57.835340 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:57.835387 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:57.839307 (+ 3920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:57.839737 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:57.839742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:57.840009 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:57.840320 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:57.840328 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:57.841190 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:57.841191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:57.842017 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:57.842019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:57.842431 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:57.842432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:57.842572 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:57.843154 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:57.843164 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:57.844739 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:20:57.847037 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:20:57.847044 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:20:57.847047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:20:57.847380 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:20:57.847643 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:20:57.847644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:20:57.847644 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:20:57.847813 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:20:57.848150 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:20:57.848157 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:20:57.848372 (+ 215us) client_negotiation.cc:769] Sending connection context
0924 00:20:57.848491 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 00:20:57.848661 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":105,"threads_started":1}
I20260924 00:20:57.849273 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:57.835173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37187 (local address 127.0.98.62:33377)
0924 00:20:57.835449 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:57.835451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:57.835512 (+ 61us) server_negotiation.cc:407] Connection header received
0924 00:20:57.839518 (+ 4006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:57.839520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:57.839537 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:57.839601 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:57.840439 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:57.841070 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:57.842130 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:57.842329 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:57.842423 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:57.844855 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:20:57.844857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:20:57.844857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:20:57.844866 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:20:57.846908 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:20:57.847490 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:20:57.847492 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:20:57.847492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:20:57.847565 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:20:57.847905 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:20:57.847906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:20:57.847906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:20:57.848066 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:20:57.848138 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:57.849083 (+ 945us) server_negotiation.cc:299] Negotiation successful
0924 00:20:57.849173 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":73,"threads_started":1}
I20260924 00:20:57.849761 11807 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33377
I20260924 00:20:57.849880 11807 heartbeater.cc:464] Registering TS with master...
I20260924 00:20:57.850104 11807 heartbeater.cc:511] Master 127.0.98.62:33377 requested a full tablet report, sending...
I20260924 00:20:57.850680 11595 ts_manager.cc:194] Registered new tserver with Master: 7c08b1d2a7cf4461b1b5a2644b372f13 (127.0.98.1:39137)
I20260924 00:20:57.851631 11595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37187
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:20:57.876468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45395/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33377
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:20:57.990217 11813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:20:57.990541 11813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:20:57.990635 11813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:20:57.992133 11813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:20:57.992228 11813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:20:57.992274 11813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:20:57.993995 11813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:45395/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:20:57.994446 11813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:20:57.994699 11813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:57.999912 11821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:57.999971 11820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:58.000331 11823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Sep 24 00:20:57 dist-test-slave-777x krb5kdc[11119](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790209257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:20:58.000998 11813 server_base.cc:1034] running on GCE node
I20260924 00:20:58.001256 11813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:20:58.001533 11813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:20:58.004827 11813 hybrid_clock.cc:648] HybridClock initialized: now 1790209258004807 us; error 32 us; skew 500 ppm
Sep 24 00:20:58 dist-test-slave-777x krb5kdc[11119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:20:58.008116 11813 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:20:58.008698 11813 webserver.cc:460] Webserver started at http://127.0.98.2:41367/ using document root <none> and password file <none>
I20260924 00:20:58.008970 11813 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:58.009042 11813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:58.009209 11813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 00:20:58 dist-test-slave-777x krb5kdc[11119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 00:20:58.021939 11813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "f62c78aa69f344d98515436de11cec00"
format_stamp: "Formatted at 2026-09-24 00:20:58 on dist-test-slave-777x"
server_key: "ffbdb6506d673ac2b67dbf84719a7011"
server_key_iv: "cb5d819ecddf464e5f52bab13fa5b7d6"
server_key_version: "kuduclusterkey@0"
I20260924 00:20:58.024487 11813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f62c78aa69f344d98515436de11cec00"
format_stamp: "Formatted at 2026-09-24 00:20:58 on dist-test-slave-777x"
server_key: "ffbdb6506d673ac2b67dbf84719a7011"
server_key_iv: "cb5d819ecddf464e5f52bab13fa5b7d6"
server_key_version: "kuduclusterkey@0"
I20260924 00:20:58.027515 11813 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 00:20:58.042960 11830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:58.043216 11813 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 00:20:58.043306 11813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "f62c78aa69f344d98515436de11cec00"
format_stamp: "Formatted at 2026-09-24 00:20:58 on dist-test-slave-777x"
server_key: "ffbdb6506d673ac2b67dbf84719a7011"
server_key_iv: "cb5d819ecddf464e5f52bab13fa5b7d6"
server_key_version: "kuduclusterkey@0"
I20260924 00:20:58.043401 11813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:58.078752 11813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:58.081386 11813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:20:58.081584 11813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:58.081843 11813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:20:58.082288 11813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:20:58.082350 11813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:58.082422 11813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:20:58.082455 11813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:58.090473 11813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42617
I20260924 00:20:58.090935 11813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:20:58.092129 11943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42617 every 8 connection(s)
I20260924 00:20:58.098376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 11813
I20260924 00:20:58.098529 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:20:58.098757 392 external_mini_cluster.cc:1428] Setting key d5979c7a474d10e89c5795ae5bb05a3b
Sep 24 00:20:58 dist-test-slave-777x krb5kdc[11119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:20:58.108831 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:58.092203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37645 (local address 127.0.98.62:33377)
0924 00:20:58.092262 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:58.092266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:58.092280 (+ 14us) server_negotiation.cc:407] Connection header received
0924 00:20:58.096738 (+ 4458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:58.096740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:58.096756 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:58.096809 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:58.097538 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.098100 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.100764 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.101148 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.101211 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:58.104491 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:20:58.104494 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:20:58.104495 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:20:58.104506 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:20:58.106752 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:20:58.107392 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:20:58.107394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:20:58.107394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:20:58.107492 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:20:58.107858 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:20:58.107859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:20:58.107859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:20:58.108025 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:20:58.108087 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:58.108580 (+ 493us) server_negotiation.cc:299] Negotiation successful
0924 00:20:58.108694 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 00:20:58.109795 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:58.091840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33377 (local address 127.0.98.2:37645)
0924 00:20:58.092083 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:58.092141 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:58.096554 (+ 4413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:58.096903 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:58.096908 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:58.097134 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:58.097352 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:58.097357 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.099005 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.099006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:58.100647 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:58.100650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.101331 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.101332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:58.101529 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:58.102219 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:58.102238 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:58.104311 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:20:58.106877 (+ 2566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:20:58.106887 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:20:58.106892 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:20:58.107273 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:20:58.107594 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:20:58.107595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:20:58.107595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:20:58.107768 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:20:58.108126 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:20:58.108133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:20:58.108373 (+ 240us) client_negotiation.cc:769] Sending connection context
0924 00:20:58.109276 (+ 903us) client_negotiation.cc:240] Negotiation successful
0924 00:20:58.109484 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":96,"threads_started":1}
I20260924 00:20:58.110630 11944 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33377
I20260924 00:20:58.110747 11944 heartbeater.cc:464] Registering TS with master...
I20260924 00:20:58.110960 11944 heartbeater.cc:511] Master 127.0.98.62:33377 requested a full tablet report, sending...
I20260924 00:20:58.111409 11595 ts_manager.cc:194] Registered new tserver with Master: f62c78aa69f344d98515436de11cec00 (127.0.98.2:42617)
I20260924 00:20:58.111954 11595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:20:58.132959 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45395/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33377
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:20:58.261301 11950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:20:58.261631 11950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:20:58.261672 11950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:20:58.263715 11950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:20:58.263800 11950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:20:58.263850 11950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:20:58.266664 11950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:45395/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:20:58.266958 11950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:20:58.267292 11950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:20:58.273375 11958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:58.273384 11955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:20:58.273577 11956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:20:58.273917 11950 server_base.cc:1034] running on GCE node
I20260924 00:20:58.274183 11950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:20:58.274402 11950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:20:58.277757 11950 hybrid_clock.cc:648] HybridClock initialized: now 1790209258277723 us; error 51 us; skew 500 ppm
Sep 24 00:20:58 dist-test-slave-777x krb5kdc[11119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:20:58.281813 11950 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:20:58.282516 11950 webserver.cc:460] Webserver started at http://127.0.98.3:35301/ using document root <none> and password file <none>
I20260924 00:20:58.282799 11950 fs_manager.cc:359] Metadata directory not provided
I20260924 00:20:58.282856 11950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:20:58.282996 11950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 00:20:58 dist-test-slave-777x krb5kdc[11119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 00:20:58.313297 11950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "5166cb3c4c1245e08bc10f85141bab6c"
format_stamp: "Formatted at 2026-09-24 00:20:58 on dist-test-slave-777x"
server_key: "c2ac4a452a13e2f7c56a2a0e43a8ed2f"
server_key_iv: "a8182b88ec8293b31abcf3091b8f0cc8"
server_key_version: "kuduclusterkey@0"
I20260924 00:20:58.313858 11950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5166cb3c4c1245e08bc10f85141bab6c"
format_stamp: "Formatted at 2026-09-24 00:20:58 on dist-test-slave-777x"
server_key: "c2ac4a452a13e2f7c56a2a0e43a8ed2f"
server_key_iv: "a8182b88ec8293b31abcf3091b8f0cc8"
server_key_version: "kuduclusterkey@0"
I20260924 00:20:58.315972 11950 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 00:20:58.332229 11965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:58.332508 11950 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:20:58.332614 11950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "5166cb3c4c1245e08bc10f85141bab6c"
format_stamp: "Formatted at 2026-09-24 00:20:58 on dist-test-slave-777x"
server_key: "c2ac4a452a13e2f7c56a2a0e43a8ed2f"
server_key_iv: "a8182b88ec8293b31abcf3091b8f0cc8"
server_key_version: "kuduclusterkey@0"
I20260924 00:20:58.332711 11950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:20:58.349102 11950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:20:58.351927 11950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:20:58.352265 11950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:20:58.352532 11950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:20:58.352873 11950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:20:58.352931 11950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:58.352974 11950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:20:58.352999 11950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:20:58.361291 11950 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46299
I20260924 00:20:58.361403 12078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46299 every 8 connection(s)
I20260924 00:20:58.361760 11950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 00:20:58 dist-test-slave-777x krb5kdc[11119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:20:58.370716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 11950
I20260924 00:20:58.370831 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:20:58.371029 392 external_mini_cluster.cc:1428] Setting key e886606f0039c8ddef4000246982c705
I20260924 00:20:58.374723 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:58.362720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49389 (local address 127.0.98.62:33377)
0924 00:20:58.362768 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:58.362772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:58.363441 (+ 669us) server_negotiation.cc:407] Connection header received
0924 00:20:58.363993 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:58.363995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:58.364008 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:58.364064 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:58.364710 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.365298 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.366756 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.367113 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.367204 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:58.370505 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:20:58.370507 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:20:58.370507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:20:58.370517 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:20:58.372945 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:20:58.373527 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:20:58.373529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:20:58.373529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:20:58.373602 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:20:58.373902 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:20:58.373903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:20:58.373904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:20:58.374048 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:20:58.374109 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:58.374528 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 00:20:58.374620 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:20:58.375396 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:58.363079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33377 (local address 127.0.98.3:49389)
0924 00:20:58.363290 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:58.363330 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:58.363873 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:58.364160 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:58.364164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:58.364405 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:20:58.364607 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:58.364611 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.365392 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.365394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:58.366609 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:58.366612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.367223 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.367224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:58.367373 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:58.368156 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:20:58.368196 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:20:58.370326 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:20:58.373067 (+ 2741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:20:58.373076 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:20:58.373080 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:20:58.373423 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:20:58.373679 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:20:58.373679 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:20:58.373680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:20:58.373825 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:20:58.374161 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:20:58.374167 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:20:58.374359 (+ 192us) client_negotiation.cc:769] Sending connection context
0924 00:20:58.374966 (+ 607us) client_negotiation.cc:240] Negotiation successful
0924 00:20:58.375157 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":104,"threads_started":1}
I20260924 00:20:58.375928 12079 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33377
I20260924 00:20:58.376032 12079 heartbeater.cc:464] Registering TS with master...
I20260924 00:20:58.376264 12079 heartbeater.cc:511] Master 127.0.98.62:33377 requested a full tablet report, sending...
I20260924 00:20:58.376655 11595 ts_manager.cc:194] Registered new tserver with Master: 5166cb3c4c1245e08bc10f85141bab6c (127.0.98.3:46299)
I20260924 00:20:58.377372 11595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49389
I20260924 00:20:58.382645 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:20:58.393190 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:58.384359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.0.98.62:33377)
0924 00:20:58.384408 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:58.384411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:58.384753 (+ 342us) server_negotiation.cc:407] Connection header received
0924 00:20:58.384859 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:58.384860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:58.384871 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:58.384903 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:20:58.385747 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.386546 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.387674 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.387967 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.388013 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:58.389757 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:20:58.389759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:20:58.389759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:20:58.389767 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:20:58.391699 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:20:58.392160 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:20:58.392161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:20:58.392162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:20:58.392264 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:20:58.392509 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:20:58.392510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:20:58.392510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:20:58.392656 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:20:58.392738 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:58.392970 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 00:20:58.393070 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:20:58.395849 11595 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:20:58.398346 11595 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:20:58.410414 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:58.402809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42617 (local address 127.0.0.1:37188)
0924 00:20:58.406029 (+ 3220us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:58.406041 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:58.406107 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:58.406378 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:58.406378 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:58.406381 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:20:58.406437 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:58.406438 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.407666 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.407667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:58.408863 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:58.408866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.409996 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.409997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:58.410120 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:58.410140 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 00:20:58.410221 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 00:20:58.410283 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3202,"thread_start_us":65,"threads_started":1}
I20260924 00:20:58.410876 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:58.403316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37188 (local address 127.0.98.2:42617)
0924 00:20:58.403611 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:58.403616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:58.406171 (+ 2555us) server_negotiation.cc:407] Connection header received
0924 00:20:58.406216 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:58.406218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:58.406234 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:58.406302 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:20:58.406511 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.407568 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.408988 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.409794 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.409870 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:58.409955 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:58.410744 (+ 789us) server_negotiation.cc:299] Negotiation successful
0924 00:20:58.410791 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":77,"threads_started":1}
I20260924 00:20:58.411940 11878 tablet_service.cc:1467] Processing CreateTablet for tablet f7ad58a6e5764e3db400ddc7a9a03da7 (DEFAULT_TABLE table=test-table [id=5f53a89e21674ff3adbed74c6b17b2a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:20:58.412410 11878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7ad58a6e5764e3db400ddc7a9a03da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:58.416790 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:58.402861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34054 (local address 127.0.98.3:46299)
0924 00:20:58.403393 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:58.403399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:58.404062 (+ 663us) server_negotiation.cc:407] Connection header received
0924 00:20:58.404239 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:58.404242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:58.404259 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:58.404315 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:20:58.404540 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.405648 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.410107 (+ 4459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.411831 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.411890 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:58.411967 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:58.416621 (+ 4654us) server_negotiation.cc:299] Negotiation successful
0924 00:20:58.416689 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":79,"threads_started":1}
I20260924 00:20:58.417279 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:58.402592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46299 (local address 127.0.0.1:34054)
0924 00:20:58.403981 (+ 1389us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:58.403992 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:58.404094 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:58.404420 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:58.404421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:58.404423 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:20:58.404466 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:58.404468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.405747 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.405748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:58.408987 (+ 3239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:58.408990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.416393 (+ 7403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.416395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:58.416489 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:58.416494 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:20:58.417117 (+ 623us) client_negotiation.cc:240] Negotiation successful
0924 00:20:58.417187 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1320,"mutex_wait_us":1460,"thread_start_us":106,"threads_started":1}
I20260924 00:20:58.418078 12013 tablet_service.cc:1467] Processing CreateTablet for tablet f7ad58a6e5764e3db400ddc7a9a03da7 (DEFAULT_TABLE table=test-table [id=5f53a89e21674ff3adbed74c6b17b2a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:20:58.418469 12013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7ad58a6e5764e3db400ddc7a9a03da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:58.422338 12099 tablet_bootstrap.cc:492] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c: Bootstrap starting.
I20260924 00:20:58.422761 12098 tablet_bootstrap.cc:492] T f7ad58a6e5764e3db400ddc7a9a03da7 P f62c78aa69f344d98515436de11cec00: Bootstrap starting.
I20260924 00:20:58.422855 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:58.402590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39137 (local address 127.0.0.1:51242)
0924 00:20:58.403070 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:58.403115 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:58.403242 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:58.403863 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:58.403867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:58.403870 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:20:58.403948 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:58.403950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.405821 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.405822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:58.407346 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:58.407349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.422516 (+ 15167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.422518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:58.422622 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:58.422629 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 00:20:58.422704 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 00:20:58.422763 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":43,"thread_start_us":77,"threads_started":1}
I20260924 00:20:58.423441 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:58.403118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51242 (local address 127.0.98.1:39137)
0924 00:20:58.403676 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:58.403679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:58.403690 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:20:58.403728 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:58.403730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:58.403748 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:58.403834 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:20:58.404056 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.405727 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.409092 (+ 3365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.409943 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.422481 (+ 12538us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:58.422591 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:58.423292 (+ 701us) server_negotiation.cc:299] Negotiation successful
0924 00:20:58.423358 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":68,"threads_started":1}
I20260924 00:20:58.424029 12098 tablet_bootstrap.cc:654] T f7ad58a6e5764e3db400ddc7a9a03da7 P f62c78aa69f344d98515436de11cec00: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:58.424476 12098 log.cc:824] T f7ad58a6e5764e3db400ddc7a9a03da7 P f62c78aa69f344d98515436de11cec00: Log is configured to *not* fsync() on all Append() calls
I20260924 00:20:58.424376 11741 tablet_service.cc:1467] Processing CreateTablet for tablet f7ad58a6e5764e3db400ddc7a9a03da7 (DEFAULT_TABLE table=test-table [id=5f53a89e21674ff3adbed74c6b17b2a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:20:58.424796 11741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7ad58a6e5764e3db400ddc7a9a03da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:20:58.424968 12099 tablet_bootstrap.cc:654] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:58.425292 12099 log.cc:824] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c: Log is configured to *not* fsync() on all Append() calls
I20260924 00:20:58.425395 12098 tablet_bootstrap.cc:492] T f7ad58a6e5764e3db400ddc7a9a03da7 P f62c78aa69f344d98515436de11cec00: No bootstrap required, opened a new log
I20260924 00:20:58.425457 12098 ts_tablet_manager.cc:1397] T f7ad58a6e5764e3db400ddc7a9a03da7 P f62c78aa69f344d98515436de11cec00: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:20:58.427511 12098 raft_consensus.cc:348] T f7ad58a6e5764e3db400ddc7a9a03da7 P f62c78aa69f344d98515436de11cec00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c08b1d2a7cf4461b1b5a2644b372f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39137 } } peers { permanent_uuid: "f62c78aa69f344d98515436de11cec00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42617 } } peers { permanent_uuid: "5166cb3c4c1245e08bc10f85141bab6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46299 } }
I20260924 00:20:58.427690 12098 raft_consensus.cc:374] T f7ad58a6e5764e3db400ddc7a9a03da7 P f62c78aa69f344d98515436de11cec00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:58.427729 12098 raft_consensus.cc:729] T f7ad58a6e5764e3db400ddc7a9a03da7 P f62c78aa69f344d98515436de11cec00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f62c78aa69f344d98515436de11cec00, State: Initialized, Role: FOLLOWER
I20260924 00:20:58.427852 12098 consensus_queue.cc:260] T f7ad58a6e5764e3db400ddc7a9a03da7 P f62c78aa69f344d98515436de11cec00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c08b1d2a7cf4461b1b5a2644b372f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39137 } } peers { permanent_uuid: "f62c78aa69f344d98515436de11cec00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42617 } } peers { permanent_uuid: "5166cb3c4c1245e08bc10f85141bab6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46299 } }
I20260924 00:20:58.428143 12098 ts_tablet_manager.cc:1428] T f7ad58a6e5764e3db400ddc7a9a03da7 P f62c78aa69f344d98515436de11cec00: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:20:58.428349 11944 heartbeater.cc:503] Master 127.0.98.62:33377 was elected leader, sending a full tablet report...
I20260924 00:20:58.429968 12102 tablet_bootstrap.cc:492] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13: Bootstrap starting.
I20260924 00:20:58.430806 12102 tablet_bootstrap.cc:654] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13: Neither blocks nor log segments found. Creating new log.
I20260924 00:20:58.431205 12102 log.cc:824] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13: Log is configured to *not* fsync() on all Append() calls
I20260924 00:20:58.432361 12102 tablet_bootstrap.cc:492] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13: No bootstrap required, opened a new log
I20260924 00:20:58.432431 12102 ts_tablet_manager.cc:1397] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:20:58.433063 12099 tablet_bootstrap.cc:492] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c: No bootstrap required, opened a new log
I20260924 00:20:58.433145 12099 ts_tablet_manager.cc:1397] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.002s
I20260924 00:20:58.434998 12102 raft_consensus.cc:348] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c08b1d2a7cf4461b1b5a2644b372f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39137 } } peers { permanent_uuid: "f62c78aa69f344d98515436de11cec00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42617 } } peers { permanent_uuid: "5166cb3c4c1245e08bc10f85141bab6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46299 } }
I20260924 00:20:58.435190 12102 raft_consensus.cc:374] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:58.435227 12102 raft_consensus.cc:729] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c08b1d2a7cf4461b1b5a2644b372f13, State: Initialized, Role: FOLLOWER
I20260924 00:20:58.435344 12099 raft_consensus.cc:348] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c08b1d2a7cf4461b1b5a2644b372f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39137 } } peers { permanent_uuid: "f62c78aa69f344d98515436de11cec00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42617 } } peers { permanent_uuid: "5166cb3c4c1245e08bc10f85141bab6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46299 } }
I20260924 00:20:58.435523 12102 consensus_queue.cc:260] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c08b1d2a7cf4461b1b5a2644b372f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39137 } } peers { permanent_uuid: "f62c78aa69f344d98515436de11cec00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42617 } } peers { permanent_uuid: "5166cb3c4c1245e08bc10f85141bab6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46299 } }
I20260924 00:20:58.435613 12099 raft_consensus.cc:374] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:20:58.435696 12099 raft_consensus.cc:729] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5166cb3c4c1245e08bc10f85141bab6c, State: Initialized, Role: FOLLOWER
I20260924 00:20:58.435945 12102 ts_tablet_manager.cc:1428] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:20:58.435904 12099 consensus_queue.cc:260] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c08b1d2a7cf4461b1b5a2644b372f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39137 } } peers { permanent_uuid: "f62c78aa69f344d98515436de11cec00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42617 } } peers { permanent_uuid: "5166cb3c4c1245e08bc10f85141bab6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46299 } }
I20260924 00:20:58.436156 11807 heartbeater.cc:503] Master 127.0.98.62:33377 was elected leader, sending a full tablet report...
I20260924 00:20:58.436239 12099 ts_tablet_manager.cc:1428] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 00:20:58.436360 12079 heartbeater.cc:503] Master 127.0.98.62:33377 was elected leader, sending a full tablet report...
I20260924 00:20:58.498023 12105 raft_consensus.cc:482] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:20:58.498209 12105 raft_consensus.cc:504] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c08b1d2a7cf4461b1b5a2644b372f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39137 } } peers { permanent_uuid: "f62c78aa69f344d98515436de11cec00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42617 } } peers { permanent_uuid: "5166cb3c4c1245e08bc10f85141bab6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46299 } }
I20260924 00:20:58.498643 12105 leader_election.cc:288] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f62c78aa69f344d98515436de11cec00 (127.0.98.2:42617), 5166cb3c4c1245e08bc10f85141bab6c (127.0.98.3:46299)
I20260924 00:20:58.502977 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:58.498963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59875 (local address 127.0.98.2:42617)
0924 00:20:58.499162 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:58.499164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:58.499174 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:20:58.499202 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:58.499203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:58.499218 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:58.499282 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:20:58.499486 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.500279 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.502004 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.502562 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.502609 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:58.502621 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:58.502864 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 00:20:58.502901 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260924 00:20:58.503324 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:58.499352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59121 (local address 127.0.98.3:46299)
0924 00:20:58.499397 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:58.499399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:58.499506 (+ 107us) server_negotiation.cc:407] Connection header received
0924 00:20:58.499600 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:58.499601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:58.499611 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:58.499640 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:20:58.499842 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.500611 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.502328 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.502873 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.502924 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:58.502934 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:58.503194 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 00:20:58.503255 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:20:58.503427 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:58.498800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42617 (local address 127.0.98.1:59875)
0924 00:20:58.499040 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:58.499052 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:58.499104 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:58.499299 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:58.499300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:58.499313 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:20:58.499399 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:58.499401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.500398 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.500398 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:58.501874 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:58.501877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.502715 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.502716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:58.502788 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:58.502791 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:20:58.503298 (+ 507us) client_negotiation.cc:240] Negotiation successful
0924 00:20:58.503343 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":96,"threads_started":1}
I20260924 00:20:58.503849 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:58.498967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46299 (local address 127.0.98.1:59121)
0924 00:20:58.499449 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:58.499461 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:20:58.499529 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:20:58.499698 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:20:58.499699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:20:58.499700 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:20:58.499771 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:58.499773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.500780 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.500781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:58.501928 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:20:58.501932 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.503027 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.503028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:20:58.503107 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:58.503111 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:20:58.503706 (+ 595us) client_negotiation.cc:240] Negotiation successful
0924 00:20:58.503751 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":79,"threads_started":1}
I20260924 00:20:58.503793 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7ad58a6e5764e3db400ddc7a9a03da7" candidate_uuid: "7c08b1d2a7cf4461b1b5a2644b372f13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f62c78aa69f344d98515436de11cec00" is_pre_election: true
I20260924 00:20:58.503980 11893 raft_consensus.cc:2454] T f7ad58a6e5764e3db400ddc7a9a03da7 P f62c78aa69f344d98515436de11cec00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c08b1d2a7cf4461b1b5a2644b372f13 in term 0.
I20260924 00:20:58.504156 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7ad58a6e5764e3db400ddc7a9a03da7" candidate_uuid: "7c08b1d2a7cf4461b1b5a2644b372f13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5166cb3c4c1245e08bc10f85141bab6c" is_pre_election: true
I20260924 00:20:58.504274 11694 leader_election.cc:302] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c08b1d2a7cf4461b1b5a2644b372f13, f62c78aa69f344d98515436de11cec00; no voters:
I20260924 00:20:58.504318 12033 raft_consensus.cc:2454] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c08b1d2a7cf4461b1b5a2644b372f13 in term 0.
I20260924 00:20:58.504424 12105 raft_consensus.cc:2781] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:20:58.504500 12105 raft_consensus.cc:482] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:20:58.504537 12105 raft_consensus.cc:3037] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:58.505434 12105 raft_consensus.cc:504] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c08b1d2a7cf4461b1b5a2644b372f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39137 } } peers { permanent_uuid: "f62c78aa69f344d98515436de11cec00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42617 } } peers { permanent_uuid: "5166cb3c4c1245e08bc10f85141bab6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46299 } }
I20260924 00:20:58.505580 12105 leader_election.cc:288] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13 [CANDIDATE]: Term 1 election: Requested vote from peers f62c78aa69f344d98515436de11cec00 (127.0.98.2:42617), 5166cb3c4c1245e08bc10f85141bab6c (127.0.98.3:46299)
I20260924 00:20:58.506983 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7ad58a6e5764e3db400ddc7a9a03da7" candidate_uuid: "7c08b1d2a7cf4461b1b5a2644b372f13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5166cb3c4c1245e08bc10f85141bab6c"
I20260924 00:20:58.507071 12033 raft_consensus.cc:3037] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:58.508401 11893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7ad58a6e5764e3db400ddc7a9a03da7" candidate_uuid: "7c08b1d2a7cf4461b1b5a2644b372f13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f62c78aa69f344d98515436de11cec00"
I20260924 00:20:58.508502 11893 raft_consensus.cc:3037] T f7ad58a6e5764e3db400ddc7a9a03da7 P f62c78aa69f344d98515436de11cec00 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:20:58.509243 11893 raft_consensus.cc:2454] T f7ad58a6e5764e3db400ddc7a9a03da7 P f62c78aa69f344d98515436de11cec00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c08b1d2a7cf4461b1b5a2644b372f13 in term 1.
I20260924 00:20:58.509469 11694 leader_election.cc:302] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c08b1d2a7cf4461b1b5a2644b372f13, f62c78aa69f344d98515436de11cec00; no voters:
I20260924 00:20:58.509572 12105 raft_consensus.cc:2781] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:20:58.509763 12105 raft_consensus.cc:686] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13 [term 1 LEADER]: Becoming Leader. State: Replica: 7c08b1d2a7cf4461b1b5a2644b372f13, State: Running, Role: LEADER
I20260924 00:20:58.509805 12033 raft_consensus.cc:2454] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c08b1d2a7cf4461b1b5a2644b372f13 in term 1.
I20260924 00:20:58.509868 12105 consensus_queue.cc:237] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c08b1d2a7cf4461b1b5a2644b372f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39137 } } peers { permanent_uuid: "f62c78aa69f344d98515436de11cec00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42617 } } peers { permanent_uuid: "5166cb3c4c1245e08bc10f85141bab6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46299 } }
I20260924 00:20:58.513336 11595 catalog_manager.cc:5495] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c08b1d2a7cf4461b1b5a2644b372f13 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7c08b1d2a7cf4461b1b5a2644b372f13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c08b1d2a7cf4461b1b5a2644b372f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f62c78aa69f344d98515436de11cec00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5166cb3c4c1245e08bc10f85141bab6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46299 } health_report { overall_health: UNKNOWN } } }
I20260924 00:20:58.526118 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:58.522683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51246 (local address 127.0.98.1:39137)
0924 00:20:58.522727 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 00:20:58.522729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:20:58.522739 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:20:58.522812 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:20:58.522812 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:20:58.522835 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:20:58.522916 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:20:58.523124 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.523871 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.525072 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:20:58.525378 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:20:58.525444 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:20:58.525679 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:20:58.525825 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:20:58.525921 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 00:20:58.525989 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 00:20:58.526015 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 00:20:58 dist-test-slave-777x krb5kdc[11119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 00:20:58.528139 12033 raft_consensus.cc:1261] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c [term 1 FOLLOWER]: Refusing update from remote peer 7c08b1d2a7cf4461b1b5a2644b372f13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:20:58.528141 11893 raft_consensus.cc:1261] T f7ad58a6e5764e3db400ddc7a9a03da7 P f62c78aa69f344d98515436de11cec00 [term 1 FOLLOWER]: Refusing update from remote peer 7c08b1d2a7cf4461b1b5a2644b372f13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:20:58.528579 12109 consensus_queue.cc:1035] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5166cb3c4c1245e08bc10f85141bab6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:58.528698 12105 consensus_queue.cc:1035] T f7ad58a6e5764e3db400ddc7a9a03da7 P 7c08b1d2a7cf4461b1b5a2644b372f13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f62c78aa69f344d98515436de11cec00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:20:58.532493 12112 mvcc.cc:205] Tried to move back new op lower bound from 7332697122928164864 to 7332697122857021440. Current Snapshot: MvccSnapshot[applied={T|T < 7332697122928164864}]
I20260924 00:20:58.535100 12113 mvcc.cc:205] Tried to move back new op lower bound from 7332697122928164864 to 7332697122857021440. Current Snapshot: MvccSnapshot[applied={T|T < 7332697122928164864}]
I20260924 00:20:58.537961 11594 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:20:58.538095 11594 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:20:58.548313 11594 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 425fb57711784a3ea0ea9f0defa62ce0: Sending DeleteTablet for 3 replicas of tablet f7ad58a6e5764e3db400ddc7a9a03da7
I20260924 00:20:58.549149 11741 tablet_service.cc:1514] Processing DeleteTablet for tablet f7ad58a6e5764e3db400ddc7a9a03da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:20:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51242
I20260924 00:20:58.549434 11878 tablet_service.cc:1514] Processing DeleteTablet for tablet f7ad58a6e5764e3db400ddc7a9a03da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:20:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37188
I20260924 00:20:58.549645 12013 tablet_service.cc:1514] Processing DeleteTablet for tablet f7ad58a6e5764e3db400ddc7a9a03da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:20:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34054
I20260924 00:20:58.550510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 11676
I20260924 00:20:58.561299 12122 tablet_replica.cc:330] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c: stopping tablet replica
I20260924 00:20:58.561473 12122 raft_consensus.cc:2229] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:20:58.561580 12122 pending_rounds.cc:62] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c: Trying to abort 2 pending ops.
I20260924 00:20:58.561615 12122 pending_rounds.cc:68] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7332697122928164864 op_type: WRITE_OP write_request { tablet_id: "f7ad58a6e5764e3db400ddc7a9a03da7" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\226\334\321\325\006\"8\n\ntest-admin\022*\n 5f53a89e21674ff3adbed74c6b17b2a3\020\001\030\001 \001(\001" signature: "<redacted>""$\340\373L\341:C\226\303%l\375\232)\226d\255L6\314\263\242v\247\356U\310\374\223\025\264k_\327H\000\335px\261\367_8Y8\004\232\364\210\006n\221\254\241\006[L\007\200\354\317\033\234\372" signing_key_seq_num: 0 } } request_id { client_id: "af4672b788f94515a543763af2918dc4" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260924 00:20:58.561867 12122 pending_rounds.cc:68] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332697122857021440 op_type: NO_OP noop_request { }
I20260924 00:20:58.561910 12122 raft_consensus.cc:2866] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260924 00:20:58.561939 12122 raft_consensus.cc:2258] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:20:58.562112 12121 tablet_replica.cc:330] T f7ad58a6e5764e3db400ddc7a9a03da7 P f62c78aa69f344d98515436de11cec00: stopping tablet replica
W20260924 00:20:58.562206 11968 connection.cc:535] server connection from 127.0.98.1:59121 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 00:20:58.562266 12121 raft_consensus.cc:2229] T f7ad58a6e5764e3db400ddc7a9a03da7 P f62c78aa69f344d98515436de11cec00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:20:58.562384 12121 raft_consensus.cc:2258] T f7ad58a6e5764e3db400ddc7a9a03da7 P f62c78aa69f344d98515436de11cec00 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 00:20:58.562577 11579 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39137 (error 108)
I20260924 00:20:58.562886 12122 ts_tablet_manager.cc:1905] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:20:58.563067 12121 ts_tablet_manager.cc:1905] T f7ad58a6e5764e3db400ddc7a9a03da7 P f62c78aa69f344d98515436de11cec00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:20:58.563784 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:58.563640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39137 (local address 127.0.0.1:51250)
0924 00:20:58.563700 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:58.563743 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 00:20:58.564050 11579 catalog_manager.cc:4584] TS 7c08b1d2a7cf4461b1b5a2644b372f13 (127.0.98.1:39137): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7ad58a6e5764e3db400ddc7a9a03da7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39137: connect: Connection refused (error 111)
I20260924 00:20:58.564383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 11813
I20260924 00:20:58.566989 12122 ts_tablet_manager.cc:1918] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:20:58.567082 12122 log.cc:1196] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/f7ad58a6e5764e3db400ddc7a9a03da7
I20260924 00:20:58.567656 12122 ts_tablet_manager.cc:1939] T f7ad58a6e5764e3db400ddc7a9a03da7 P 5166cb3c4c1245e08bc10f85141bab6c: Deleting consensus metadata
I20260924 00:20:58.568439 11582 catalog_manager.cc:4841] TS 5166cb3c4c1245e08bc10f85141bab6c (127.0.98.3:46299): tablet f7ad58a6e5764e3db400ddc7a9a03da7 (table test-table [id=5f53a89e21674ff3adbed74c6b17b2a3]) successfully deleted
I20260924 00:20:58.576746 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 11950
I20260924 00:20:58.579445 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:20:58.579305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42617 (local address 127.0.0.1:37200)
0924 00:20:58.579369 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 00:20:58.579400 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 00:20:58.582449 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 11563
I20260924 00:20:58.589401 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 00:20:58.934674 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 00:20:58.934780 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 00:20:59.280411 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 00:20:59.280 UTC [11149] LOG: received smart shutdown request
2026-09-24 00:20:59.284 UTC [11149] LOG: background worker "logical replication launcher" (PID 11158) exited with exit code 1
2026-09-24 00:20:59.285 UTC [11153] LOG: shutting down
2026-09-24 00:20:59.498 UTC [11149] LOG: database system is shut down
2026-09-24T00:20:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49248 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:20:59 dist-test-slave-777x krb5kdc[12132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:20:59 dist-test-slave-777x krb5kdc[12132](info): set up 2 sockets
Sep 24 00:20:59 dist-test-slave-777x krb5kdc[12132](info): commencing operation
krb5kdc: starting...
W20260924 00:21:00.724086 392 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:21:00 dist-test-slave-777x krb5kdc[12132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:21:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:21:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790209154318147-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790209154318147-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790209154318147-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790209154318147-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790209154318147-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790209154318147-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 00:21:00.899384 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790209154318147-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskYEWBIr/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790209154318147-392-0/minicluster-data/postgres -l logfile start
2026-09-24 00:21:03.752 UTC [12162] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 00:21:03.752 UTC [12162] LOG: listening on IPv4 address "127.0.98.20", port 36399
2026-09-24 00:21:03.756 UTC [12162] LOG: listening on Unix socket "/tmp/.s.PGSQL.36399"
2026-09-24 00:21:03.777 UTC [12165] LOG: database system was shut down at 2026-09-24 00:21:01 UTC
2026-09-24 00:21:03.781 UTC [12162] LOG: database system is ready to accept connections
I20260924 00:21:04.756130 392 mini_postgres.cc:96] Postgres bound to 36399
2026-09-24 00:21:04.762 UTC [12173] FATAL: database "slave" does not exist
127.0.98.20:36399 - accepting connections
I20260924 00:21:04.763170 392 mini_ranger.cc:161] Starting Ranger...
I20260924 00:21:04.778630 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 00:21:05.362857 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 00:21:05.362985 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790209154318147-392-0/minicluster-data/ranger-admin
2026-09-24 00:21:05,682 [I] DB FLAVOR :POSTGRES
2026-09-24 00:21:05,682 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 00:21:05,682 [I] Checking connection..
2026-09-24 00:21:05,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 00:21:05,921 [I] Checking connection passed.
2026-09-24 00:21:05,921 [I] --------- Verifying version history table ---------
2026-09-24 00:21:05,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 00:21:06,159 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 00:21:06,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 00:21:06,431 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 00:21:06,431 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 00:21:06,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36399/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 00:21:06.698 UTC [12237] WARNING: there is no transaction in progress
2026-09-24 00:21:06,704 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 00:21:06,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 00:21:06,980 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 00:21:06,980 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 00:21:06,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 00:21:07,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 00:21:07,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-777x.c.gcp-upstream.internal','N') ;"
2026-09-24 00:21:07,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 00:21:07,989 [I] Table x_portal_user does not exist in database ranger
2026-09-24 00:21:07,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 00:21:08,282 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 00:21:08,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 00:21:08,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 00:21:08,831 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 00:21:08,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36399/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 00:21:09.918 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:21:09.937 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:21:09.956 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:21:09.975 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:21:09.993 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:21:10.014 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:21:10.078 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:21:10.086 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:21:10.105 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:21:10.116 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:21:10.128 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:21:10.144 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:21:10.156 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:21:10.163 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:21:10.172 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:21:10.180 UTC [12345] WARNING: there is no transaction in progress
2026-09-24 00:21:10,974 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 00:21:10,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 00:21:11,245 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 00:21:11,246 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 00:21:11,495 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 00:21:11,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 00:21:11,749 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 00:21:11,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 00:21:11,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 00:21:12,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-777x.c.gcp-upstream.internal';"
2026-09-24 00:21:12,500 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 00:21:12,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:36399/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 00:21:12,795 [I] DB_PATCHES have already been applied
I20260924 00:21:12.801398 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790209154318147-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskYEWBIr/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskYEWBIr/build/release/bin/postgresql.jar:/tmp/dist-test-taskYEWBIr/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 00:21:12.801497 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 00:21:12.804723 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:37187
Sep 24, 2026 12:21:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 12:21:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 12:21:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790209154318147-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 12:21:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790209154318147-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 12:21:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790209154318147-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 12:21:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790209154318147-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 12:21:13 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Sep 24, 2026 12:21:42 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 12:21:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 12:21:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 12:21:42 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 12:21:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 12:21:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 12:21:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 12:21:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 12:21:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 12:21:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 12:21:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 12:21:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 12:21:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 12:21:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 12:21:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 12:21:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 12:21:42 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 12:21:43 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 00:23:13.303959 392 mini_ranger.cc:160] Time spent starting Ranger: real 128.541s user 0.000s sys 0.145s
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:1024: Failure
Failed
Bad status: Timed out: Failed to start the Ranger service: process with PID 12432 is not yet bound to any port at the specified addresses; last attempt running lsof returned 'Not found: could not find pattern of a bound port in lsof output: p12432
f310
n127.0.0.1:48690->127.0.98.20:36399
f311
n127.0.0.1:48692->127.0.98.20:36399
f312
n127.0.0.1:48704->127.0.98.20:36399
f313
n127.0.0.1:48718->127.0.98.20:36399
f314
n127.0.0.1:48724->127.0.98.20:36399'
I20260924 00:23:13.304837 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 00:23:13.335721 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 00:23:13.335 UTC [12162] LOG: received smart shutdown request
2026-09-24 00:23:13.340 UTC [12162] LOG: background worker "logical replication launcher" (PID 12171) exited with exit code 1
2026-09-24 00:23:13.340 UTC [12166] LOG: shutting down
2026-09-24 00:23:13.544 UTC [12162] LOG: database system is shut down
2026-09-24T00:23:13Z chronyd exiting
I20260924 00:23:13.560205 392 test_util.cc:175] -----------------------------------------------
I20260924 00:23:13.560280 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790209154318147-392-0
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (133933 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:23:13 dist-test-slave-777x krb5kdc[12551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:23:13 dist-test-slave-777x krb5kdc[12551](info): set up 2 sockets
Sep 24 00:23:13 dist-test-slave-777x krb5kdc[12551](info): commencing operation
krb5kdc: starting...
W20260924 00:23:14.675690 392 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:23:14 dist-test-slave-777x krb5kdc[12551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:23:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:23:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:23:14.803213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43183
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43183,127.0.98.61:39449,127.0.98.60:46243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:23:14.894472 12564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:14.894765 12564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:14.894809 12564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:14.896160 12564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:23:14.896275 12564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:14.896301 12564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:14.896325 12564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:23:14.896420 12564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:23:14.898307 12564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43183,127.0.98.61:39449,127.0.98.60:46243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:14.898635 12564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:14.898893 12564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:14.904657 12570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:14.904742 12569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:14.904808 12572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:14.904808 12564 server_base.cc:1034] running on GCE node
I20260924 00:23:14.905278 12564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:14.905614 12564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:14.906795 12564 hybrid_clock.cc:648] HybridClock initialized: now 1790209394906769 us; error 42 us; skew 500 ppm
Sep 24 00:23:14 dist-test-slave-777x krb5kdc[12551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:14.909585 12564 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:23:14.910106 12564 webserver.cc:460] Webserver started at http://127.0.98.62:42875/ using document root <none> and password file <none>
I20260924 00:23:14.910331 12564 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:14.910395 12564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:14.910549 12564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:23:14.911674 12564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "e4f14c28cbaa43589eb74fb531aeae7c"
format_stamp: "Formatted at 2026-09-24 00:23:14 on dist-test-slave-777x"
server_key: "211532f0c78d6b67db61fbe3c7b74e5e"
server_key_iv: "9251dc03bc7afbc43602380a68ebdc98"
server_key_version: "encryptionkey@0"
I20260924 00:23:14.912097 12564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "e4f14c28cbaa43589eb74fb531aeae7c"
format_stamp: "Formatted at 2026-09-24 00:23:14 on dist-test-slave-777x"
server_key: "211532f0c78d6b67db61fbe3c7b74e5e"
server_key_iv: "9251dc03bc7afbc43602380a68ebdc98"
server_key_version: "encryptionkey@0"
I20260924 00:23:14.913497 12564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:23:14.914304 12579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:14.914503 12564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:23:14.914601 12564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "e4f14c28cbaa43589eb74fb531aeae7c"
format_stamp: "Formatted at 2026-09-24 00:23:14 on dist-test-slave-777x"
server_key: "211532f0c78d6b67db61fbe3c7b74e5e"
server_key_iv: "9251dc03bc7afbc43602380a68ebdc98"
server_key_version: "encryptionkey@0"
I20260924 00:23:14.914680 12564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:14.927189 12564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:14.929678 12564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:14.929841 12564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:14.933946 12564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43183
I20260924 00:23:14.933987 12631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43183 every 8 connection(s)
I20260924 00:23:14.934376 12564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:23:14.935324 12632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:23:14.936995 12632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:23:14.938513 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 12564
I20260924 00:23:14.938637 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:23:14.938833 392 external_mini_cluster.cc:1428] Setting key 0b3f18daeda7414df14bd1c9ed9d6474
Sep 24 00:23:14 dist-test-slave-777x krb5kdc[12551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260924 00:23:14.948083 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:14.937674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43183 (local address 127.0.0.1:40692)
0924 00:23:14.937833 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:14.937874 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:14.938377 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:14.938757 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:14.938761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:14.939173 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:14.939415 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:14.939422 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:14.940796 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:14.940796 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:14.941728 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:14.941731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:14.942486 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:14.942487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:14.942574 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:14.943115 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:14.943124 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:14.944554 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:14.946719 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:14.946726 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:14.946728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:14.946917 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:14.947134 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:14.947134 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:14.947135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:14.947211 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:14.947544 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:14.947548 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:14.947619 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 00:23:14.947718 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 00:23:14.947916 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":67,"thread_start_us":81,"threads_started":1}
I20260924 00:23:14.948087 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:14.937843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40692 (local address 127.0.98.62:43183)
0924 00:23:14.938008 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:14.938012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:14.938046 (+ 34us) server_negotiation.cc:407] Connection header received
0924 00:23:14.938574 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:14.938582 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:14.938599 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:14.938695 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:14.939532 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:14.940688 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:14.941813 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:14.942373 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:14.942499 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:14.944641 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:14.944649 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:14.944650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:14.944659 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:14.946613 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:14.946992 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:14.946994 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:14.946995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:14.947073 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:14.947263 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:14.947263 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:14.947264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:14.947481 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:14.947601 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:14.947746 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 00:23:14.947845 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":42,"threads_started":1}
I20260924 00:23:14.948921 12632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260924 00:23:14.949395 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:14.949267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39449 (local address 127.0.0.1:45370)
0924 00:23:14.949314 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:14.949361 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 00:23:14.949534 12581 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39449: connect: Connection refused (error 111)
I20260924 00:23:14.949962 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:14.949857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39449 (local address 127.0.0.1:45386)
0924 00:23:14.949904 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:14.949933 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 00:23:14.950100 12632 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39449: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39449: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 00:23:14.967386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39449
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43183,127.0.98.61:39449,127.0.98.60:46243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:23:14.999329 12632 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } attempt: 1
I20260924 00:23:14.999828 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:14.999699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39449 (local address 127.0.0.1:45396)
0924 00:23:14.999762 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:14.999801 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 00:23:15.000219 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.000094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39449 (local address 127.0.0.1:45404)
0924 00:23:15.000139 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:15.000192 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 00:23:15.000312 12632 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39449: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39449: connect: Connection refused (error 111)
W20260924 00:23:15.049611 12639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:15.049842 12639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:15.049865 12639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:15.051149 12639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:23:15.051224 12639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:15.051247 12639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:15.051270 12639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:23:15.051296 12639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:23:15.053017 12639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43183,127.0.98.61:39449,127.0.98.60:46243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:15.053303 12639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:15.053534 12639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:15.058458 12645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:15.058460 12644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:15.058458 12647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:15.058715 12639 server_base.cc:1034] running on GCE node
I20260924 00:23:15.059087 12639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:15.059464 12639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:15.060643 12639 hybrid_clock.cc:648] HybridClock initialized: now 1790209395060618 us; error 42 us; skew 500 ppm
Sep 24 00:23:15 dist-test-slave-777x krb5kdc[12551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:15.063385 12639 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 00:23:15.063853 12639 webserver.cc:460] Webserver started at http://127.0.98.61:34519/ using document root <none> and password file <none>
I20260924 00:23:15.064087 12639 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:15.064137 12639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:15.064303 12639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:23:15.065439 12639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-1/data/instance:
uuid: "e9bf238337004f22b55d2272884ed4c7"
format_stamp: "Formatted at 2026-09-24 00:23:15 on dist-test-slave-777x"
server_key: "d1cfcd8d46b33b6384fddf4eebadbc70"
server_key_iv: "2d6aaeb97148898f976e0e3e8bfcf90c"
server_key_version: "encryptionkey@0"
I20260924 00:23:15.065806 12639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-1/wal/instance:
uuid: "e9bf238337004f22b55d2272884ed4c7"
format_stamp: "Formatted at 2026-09-24 00:23:15 on dist-test-slave-777x"
server_key: "d1cfcd8d46b33b6384fddf4eebadbc70"
server_key_iv: "2d6aaeb97148898f976e0e3e8bfcf90c"
server_key_version: "encryptionkey@0"
I20260924 00:23:15.067248 12639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:23:15.067991 12654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:15.068214 12639 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:23:15.068300 12639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-1/wal
uuid: "e9bf238337004f22b55d2272884ed4c7"
format_stamp: "Formatted at 2026-09-24 00:23:15 on dist-test-slave-777x"
server_key: "d1cfcd8d46b33b6384fddf4eebadbc70"
server_key_iv: "2d6aaeb97148898f976e0e3e8bfcf90c"
server_key_version: "encryptionkey@0"
I20260924 00:23:15.068357 12639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:15.068432 12632 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } attempt: 2
I20260924 00:23:15.068872 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.068737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39449 (local address 127.0.0.1:45412)
0924 00:23:15.068803 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:15.068836 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 00:23:15.069324 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.069166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39449 (local address 127.0.0.1:45422)
0924 00:23:15.069243 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:15.069297 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 00:23:15.069458 12632 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39449: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39449: connect: Connection refused (error 111)
I20260924 00:23:15.091606 12639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:15.094436 12639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:15.094632 12639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:15.099064 12639 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39449
I20260924 00:23:15.099103 12706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39449 every 8 connection(s)
I20260924 00:23:15.099514 12639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-1/data/info.pb
I20260924 00:23:15.100687 12707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:23:15.102397 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 12639
I20260924 00:23:15.102521 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-1/wal/instance
I20260924 00:23:15.102741 392 external_mini_cluster.cc:1428] Setting key fbe5e7a76c991149aed7f564c187965a
I20260924 00:23:15.102715 12707 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:23:15 dist-test-slave-777x krb5kdc[12551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260924 00:23:15.113586 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.103711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40696 (local address 127.0.98.62:43183)
0924 00:23:15.103789 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:15.103791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:15.104069 (+ 278us) server_negotiation.cc:407] Connection header received
0924 00:23:15.104730 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:15.104731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:15.104746 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:15.104828 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:15.105611 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.106511 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.107780 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.108213 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.108305 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:15.110495 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:15.110497 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:15.110498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:15.110507 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:15.112084 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.112578 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.112579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.112579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.112644 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.112880 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.112881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.112881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.113006 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:15.113073 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:15.113409 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 00:23:15.113485 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 00:23:15.113709 12709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.103691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43183 (local address 127.0.0.1:40696)
0924 00:23:15.103906 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:15.103968 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:15.104625 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:15.105027 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:15.105030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:15.105300 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:15.105513 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:15.105517 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.106636 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.106637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:15.107680 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:15.107683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.108309 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.108310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:15.108431 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:15.109015 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:15.109024 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:15.110405 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:15.112224 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:15.112233 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:15.112236 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:15.112490 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:15.112713 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:15.112714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:15.112714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:15.112825 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:15.113103 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:15.113109 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:15.113290 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 00:23:15.113382 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 00:23:15.113513 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":162,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260924 00:23:15.114372 12707 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:23:15 dist-test-slave-777x krb5kdc[12551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:23:15.122736 12711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.114698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45428 (local address 127.0.98.61:39449)
0924 00:23:15.114846 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:15.114851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:15.114869 (+ 18us) server_negotiation.cc:407] Connection header received
0924 00:23:15.114897 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:15.114898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:15.114909 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:15.114971 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:15.115330 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.116441 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.117172 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.117605 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.117674 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:15.119782 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:15.119791 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:15.119792 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:15.119798 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:15.121469 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.121864 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.121867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.121868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.121923 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.122150 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.122151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.122151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.122297 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:15.122379 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:15.122497 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 00:23:15.122602 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260924 00:23:15.122743 12709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.114659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39449 (local address 127.0.0.1:45428)
0924 00:23:15.114718 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:15.114732 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:15.114781 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:15.114989 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:15.114991 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:15.115179 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:15.115248 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:15.115249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.116520 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.116521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:15.117067 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:15.117069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.117680 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.117681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:15.117776 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:15.118203 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:15.118208 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:15.119684 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:15.121604 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:15.121605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:15.121606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:15.121772 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:15.121988 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:15.121989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:15.121990 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:15.122078 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:15.122369 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:15.122369 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:15.122406 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 00:23:15.122469 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 00:23:15.122653 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":93}
I20260924 00:23:15.123414 12707 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:23:15.123790 12709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.123674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46243 (local address 127.0.0.1:58178)
0924 00:23:15.123721 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:15.123768 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 00:23:15.123926 12657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46243: connect: Connection refused (error 111)
I20260924 00:23:15.124336 12709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.124247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46243 (local address 127.0.0.1:58186)
0924 00:23:15.124289 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:15.124313 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 00:23:15.124441 12707 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46243: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46243: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 00:23:15.131278 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46243
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43183,127.0.98.61:39449,127.0.98.60:46243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:23:15.160614 12632 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } attempt: 3
Sep 24 00:23:15 dist-test-slave-777x krb5kdc[12551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:23:15.168215 12711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.161152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45444 (local address 127.0.98.61:39449)
0924 00:23:15.161216 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:15.161218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:15.161248 (+ 30us) server_negotiation.cc:407] Connection header received
0924 00:23:15.161275 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:15.161276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:15.161293 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:15.161367 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:15.161710 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.162282 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.163002 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.163216 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.163302 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:15.165520 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:15.165522 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:15.165522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:15.165531 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:15.167016 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.167383 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.167385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.167385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.167462 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.167655 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.167656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.167656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.167772 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:15.167846 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:15.168003 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 00:23:15.168088 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 00:23:15.168215 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.161080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39449 (local address 127.0.0.1:45444)
0924 00:23:15.161143 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:15.161153 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:15.161211 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:15.161376 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:15.161377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:15.161557 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:15.161629 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:15.161631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.162387 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.162388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:15.162915 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:15.162917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.163302 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.163303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:15.163385 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:15.163922 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:15.163928 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:15.165407 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:15.167130 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:15.167131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:15.167132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:15.167310 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:15.167528 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:15.167529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:15.167530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:15.167594 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:15.167838 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:15.167839 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:15.167890 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 00:23:15.167977 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 00:23:15.168086 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 00:23:15.168956 12632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:23:15.169318 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.169208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46243 (local address 127.0.0.1:58192)
0924 00:23:15.169262 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:15.169294 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 00:23:15.169695 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.169574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46243 (local address 127.0.0.1:58200)
0924 00:23:15.169639 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:15.169672 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 00:23:15.169821 12632 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46243: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46243: connect: Connection refused (error 111)
I20260924 00:23:15.173658 12707 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } attempt: 1
I20260924 00:23:15.174189 12709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.173967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46243 (local address 127.0.0.1:58208)
0924 00:23:15.174125 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:15.174161 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260924 00:23:15.174559 12709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.174474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46243 (local address 127.0.0.1:58214)
0924 00:23:15.174515 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:15.174542 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 00:23:15.174659 12707 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46243: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46243: connect: Connection refused (error 111)
I20260924 00:23:15.200984 12632 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } attempt: 1
I20260924 00:23:15.201560 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.201402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46243 (local address 127.0.0.1:58226)
0924 00:23:15.201469 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:15.201515 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 00:23:15.201992 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.201869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46243 (local address 127.0.0.1:58232)
0924 00:23:15.201935 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:15.201966 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 00:23:15.202131 12632 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46243: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46243: connect: Connection refused (error 111)
W20260924 00:23:15.219036 12714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:15.219259 12714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:15.219288 12714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:15.220502 12714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:23:15.220551 12714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:15.220568 12714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:15.220583 12714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:23:15.220597 12714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:23:15.222128 12714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43183,127.0.98.61:39449,127.0.98.60:46243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:15.222335 12714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:15.222529 12714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:15.227499 12722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:15.227532 12720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:15.227739 12714 server_base.cc:1034] running on GCE node
W20260924 00:23:15.227521 12719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:15.228092 12714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:15.228457 12714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:15.229600 12714 hybrid_clock.cc:648] HybridClock initialized: now 1790209395229589 us; error 30 us; skew 500 ppm
Sep 24 00:23:15 dist-test-slave-777x krb5kdc[12551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:15.232056 12714 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 00:23:15.232561 12714 webserver.cc:460] Webserver started at http://127.0.98.60:40129/ using document root <none> and password file <none>
I20260924 00:23:15.232805 12714 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:15.232872 12714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:15.233018 12714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:23:15.234130 12714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-2/data/instance:
uuid: "b5b8306002ce46d5b7b10f583df1bce2"
format_stamp: "Formatted at 2026-09-24 00:23:15 on dist-test-slave-777x"
server_key: "8ad0eba6462fd7ff2d89aa7af1955580"
server_key_iv: "d433ef88b38b84b6267aeba722d485be"
server_key_version: "encryptionkey@0"
I20260924 00:23:15.234508 12714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-2/wal/instance:
uuid: "b5b8306002ce46d5b7b10f583df1bce2"
format_stamp: "Formatted at 2026-09-24 00:23:15 on dist-test-slave-777x"
server_key: "8ad0eba6462fd7ff2d89aa7af1955580"
server_key_iv: "d433ef88b38b84b6267aeba722d485be"
server_key_version: "encryptionkey@0"
I20260924 00:23:15.235862 12714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:23:15.236668 12729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:15.236830 12714 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:15.236917 12714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-2/wal
uuid: "b5b8306002ce46d5b7b10f583df1bce2"
format_stamp: "Formatted at 2026-09-24 00:23:15 on dist-test-slave-777x"
server_key: "8ad0eba6462fd7ff2d89aa7af1955580"
server_key_iv: "d433ef88b38b84b6267aeba722d485be"
server_key_version: "encryptionkey@0"
I20260924 00:23:15.236994 12714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:15.242806 12707 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } attempt: 2
I20260924 00:23:15.243247 12709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.243124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46243 (local address 127.0.0.1:58236)
0924 00:23:15.243177 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:15.243218 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 00:23:15.243628 12709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.243516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46243 (local address 127.0.0.1:58248)
0924 00:23:15.243562 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:15.243598 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 00:23:15.243757 12707 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:46243: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46243: connect: Connection refused (error 111)
I20260924 00:23:15.266253 12714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:15.268375 12714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:15.268497 12714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:15.273169 12714 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46243
I20260924 00:23:15.273198 12781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46243 every 8 connection(s)
I20260924 00:23:15.273651 12714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-2/data/info.pb
I20260924 00:23:15.274582 12782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:23:15.276237 12782 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:23:15.276662 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 12714
I20260924 00:23:15.276777 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790209154318147-392-0/minicluster-data/master-2/wal/instance
I20260924 00:23:15.277000 392 external_mini_cluster.cc:1428] Setting key a0fac18c6c05fdd507a38050dbbf7faa
I20260924 00:23:15.277292 12632 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } attempt: 2
Sep 24 00:23:15 dist-test-slave-777x krb5kdc[12551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 00:23:15 dist-test-slave-777x krb5kdc[12551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 00:23:15 dist-test-slave-777x krb5kdc[12551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 00:23:15.287746 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.277698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46243 (local address 127.0.0.1:58274)
0924 00:23:15.277749 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:15.277757 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:15.277788 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:15.278220 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:15.278221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:15.278431 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:15.278490 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:15.278492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.280308 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.280309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:15.280814 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:15.280815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.281486 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.281487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:15.281577 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:15.282145 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:15.282151 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:15.283568 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:15.286045 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:15.286046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:15.286047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:15.286375 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:15.286679 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:15.286680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:15.286680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:15.286756 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:15.287316 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:15.287317 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:15.287370 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 00:23:15.287510 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 00:23:15.287625 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 00:23:15.288371 12785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.277742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58274 (local address 127.0.98.60:46243)
0924 00:23:15.277993 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:15.277995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:15.278001 (+ 6us) server_negotiation.cc:407] Connection header received
0924 00:23:15.278069 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:15.278077 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:15.278102 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:15.278679 (+ 577us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:15.278765 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.280158 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.280921 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.281402 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.281464 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:15.283676 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:15.283684 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:15.283684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:15.283691 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:15.285972 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.286474 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.286478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.286478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.286593 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.286831 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.286832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.286833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.287236 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:15.287337 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:15.287591 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 00:23:15.287747 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":213,"server-negotiator.queue_time_us":227,"thread_start_us":67,"threads_started":1}
I20260924 00:23:15.288843 12787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.277429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58258 (local address 127.0.98.60:46243)
0924 00:23:15.277911 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:15.277916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:15.277970 (+ 54us) server_negotiation.cc:407] Connection header received
0924 00:23:15.278069 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:15.278077 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:15.278102 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:15.279086 (+ 984us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:15.279114 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.280151 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.281192 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.281461 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.281578 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:15.285080 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:15.285082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:15.285082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:15.285091 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:15.287116 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.287856 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.287858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.287858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.287919 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.288124 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.288125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.288125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.288299 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:15.288450 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:15.288602 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 00:23:15.288714 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":451,"thread_start_us":64,"threads_started":1}
I20260924 00:23:15.289635 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.277052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43183 (local address 127.0.0.1:40704)
0924 00:23:15.277249 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:15.277302 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:15.277996 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:15.278360 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:15.278366 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:15.278683 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:15.278954 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:15.278959 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.279719 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.279720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:15.280765 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:15.280768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.281287 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.281288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:15.281411 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:15.281987 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:15.281996 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:15.285558 (+ 3562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:15.287634 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:15.287641 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:15.287644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:15.287946 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:15.288773 (+ 827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:15.288774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:15.288775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:15.288862 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:15.289225 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:15.289230 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:15.289292 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 00:23:15.289395 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 00:23:15.289510 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
I20260924 00:23:15.289669 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.277053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40704 (local address 127.0.98.62:43183)
0924 00:23:15.277108 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:15.277110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:15.277391 (+ 281us) server_negotiation.cc:407] Connection header received
0924 00:23:15.278094 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:15.278095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:15.278105 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:15.278325 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:15.279054 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.279612 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.280877 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.281159 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.281221 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:15.285674 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:15.285675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:15.285676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:15.285684 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:15.287555 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.288226 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.288227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.288227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.288284 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.288977 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.288979 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.288980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.289134 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:15.289230 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:15.289450 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 00:23:15.289551 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 00:23:15.290624 12782 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:23:15.290863 12632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c: Bootstrap starting.
I20260924 00:23:15.291602 12632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c: Neither blocks nor log segments found. Creating new log.
I20260924 00:23:15.291939 12632 log.cc:824] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:15.292889 12632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c: No bootstrap required, opened a new log
Sep 24 00:23:15 dist-test-slave-777x krb5kdc[12551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:23:15.295282 12632 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f14c28cbaa43589eb74fb531aeae7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } } peers { permanent_uuid: "e9bf238337004f22b55d2272884ed4c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } } peers { permanent_uuid: "b5b8306002ce46d5b7b10f583df1bce2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } }
I20260924 00:23:15.295444 12632 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:23:15.295531 12632 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4f14c28cbaa43589eb74fb531aeae7c, State: Initialized, Role: FOLLOWER
I20260924 00:23:15.295693 12632 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f14c28cbaa43589eb74fb531aeae7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } } peers { permanent_uuid: "e9bf238337004f22b55d2272884ed4c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } } peers { permanent_uuid: "b5b8306002ce46d5b7b10f583df1bce2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } }
I20260924 00:23:15.296254 12790 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f14c28cbaa43589eb74fb531aeae7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } } peers { permanent_uuid: "e9bf238337004f22b55d2272884ed4c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } } peers { permanent_uuid: "b5b8306002ce46d5b7b10f583df1bce2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } } }
I20260924 00:23:15.296401 12790 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:23:15.296317 12632 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 00:23:15 dist-test-slave-777x krb5kdc[12551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:23:15.298175 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.290891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39449 (local address 127.0.0.1:45452)
0924 00:23:15.290930 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:15.290943 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:15.290987 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:15.291188 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:15.291189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:15.291327 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:15.291380 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:15.291382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.292103 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.292104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:15.292797 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:15.292799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.293167 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.293168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:15.293241 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:15.293680 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:15.293685 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:15.294831 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:15.296817 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:15.296817 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:15.296818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:15.297084 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:15.297333 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:15.297334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:15.297334 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:15.297394 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:15.297770 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:15.297771 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:15.297842 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 00:23:15.297922 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 00:23:15.298042 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 00:23:15.298648 12801 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:23:15.298763 12801 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:23:15.298877 12711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.291008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45452 (local address 127.0.98.61:39449)
0924 00:23:15.291074 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:15.291076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:15.291084 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:23:15.291112 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:15.291113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:15.291122 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:15.291184 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:15.291466 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.291998 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.292874 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.293085 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.293169 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:15.294920 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:15.294921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:15.294922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:15.294930 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:15.296694 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.297182 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.297184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.297184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.297245 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.297474 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.297476 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.297476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.297645 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:15.297732 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:15.298621 (+ 889us) server_negotiation.cc:299] Negotiation successful
0924 00:23:15.298710 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 00:23:15.299388 12782 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:23:15.299705 12564 master_runner.cc:186] Error getting master registration for 127.0.98.61:39449: OK, instance_id { permanent_uuid: "e9bf238337004f22b55d2272884ed4c7" instance_seqno: 1790209395097867 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 00:23:15.302244 12564 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 00:23:15.302752 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.289560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40716 (local address 127.0.98.62:43183)
0924 00:23:15.289714 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:15.289750 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:15.292346 (+ 2596us) server_negotiation.cc:407] Connection header received
0924 00:23:15.292423 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:15.292424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:15.292435 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:15.292542 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:15.293210 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.294019 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.295130 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.295452 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.295787 (+ 335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:15.298436 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:15.298438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:15.298438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:15.298445 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:15.301063 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.301579 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.301581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.301581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.301644 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.301842 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.301842 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.301843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.301972 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:15.302155 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:15.302564 (+ 409us) server_negotiation.cc:299] Negotiation successful
0924 00:23:15.302648 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
Sep 24 00:23:15 dist-test-slave-777x krb5kdc[12551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 00:23:15 dist-test-slave-777x krb5kdc[12551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209394, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:23:15.309494 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.299655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46243 (local address 127.0.0.1:58280)
0924 00:23:15.299732 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:15.299743 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:15.299782 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:15.300425 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:15.300426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:15.300590 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:15.300640 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:15.300642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.301642 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.301643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:15.302428 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:15.302431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.302949 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.302950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:15.302997 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:15.303462 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:15.303468 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:15.305012 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:15.308074 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:15.308075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:15.308076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:15.308375 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:15.308601 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:15.308602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:15.308603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:15.308671 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:15.309075 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:15.309077 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:15.309140 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 00:23:15.309225 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 00:23:15.309347 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 00:23:15.309556 12787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.299865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58280 (local address 127.0.98.60:46243)
0924 00:23:15.300295 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:15.300298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:15.300307 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:23:15.300329 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:15.300330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:15.300340 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:15.300704 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:15.300725 (+ 21us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.301523 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.302564 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.302856 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.303163 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:15.305990 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:15.305991 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:15.305991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:15.305999 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:15.307946 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.308456 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.308458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.308458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.308522 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.308743 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.308744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.308744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.308972 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:15.309054 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:15.309277 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 00:23:15.309414 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":408}
I20260924 00:23:15.311120 12782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2: Bootstrap starting.
I20260924 00:23:15.311904 12782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2: Neither blocks nor log segments found. Creating new log.
I20260924 00:23:15.312256 12782 log.cc:824] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:15.312299 12711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.303326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45454 (local address 127.0.98.61:39449)
0924 00:23:15.303375 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:15.303377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:15.303412 (+ 35us) server_negotiation.cc:407] Connection header received
0924 00:23:15.303489 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:15.303490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:15.303503 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:15.303562 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:15.304212 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.305018 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.306102 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.306328 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.306448 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:15.309315 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:15.309316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:15.309317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:15.309338 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:15.311077 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.311460 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.311462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.311462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.311520 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.311739 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.311739 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.311740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.311861 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:15.312051 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:15.312090 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 00:23:15.312168 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:23:15.313164 12782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2: No bootstrap required, opened a new log
I20260924 00:23:15.314246 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:23:15.315207 12782 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f14c28cbaa43589eb74fb531aeae7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } } peers { permanent_uuid: "e9bf238337004f22b55d2272884ed4c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } } peers { permanent_uuid: "b5b8306002ce46d5b7b10f583df1bce2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } }
I20260924 00:23:15.315423 12782 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:23:15.315481 12782 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b8306002ce46d5b7b10f583df1bce2, State: Initialized, Role: FOLLOWER
I20260924 00:23:15.315630 12782 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f14c28cbaa43589eb74fb531aeae7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } } peers { permanent_uuid: "e9bf238337004f22b55d2272884ed4c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } } peers { permanent_uuid: "b5b8306002ce46d5b7b10f583df1bce2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } }
I20260924 00:23:15.316226 12782 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:23:15.316293 12809 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f14c28cbaa43589eb74fb531aeae7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } } peers { permanent_uuid: "e9bf238337004f22b55d2272884ed4c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } } peers { permanent_uuid: "b5b8306002ce46d5b7b10f583df1bce2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } } }
I20260924 00:23:15.316377 12809 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:23:15.320096 12714 master_runner.cc:186] Error getting master registration for 127.0.98.61:39449: OK, instance_id { permanent_uuid: "e9bf238337004f22b55d2272884ed4c7" instance_seqno: 1790209395097867 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 00:23:15.323206 12714 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 00:23:15.325014 12788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.315854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40730 (local address 127.0.98.62:43183)
0924 00:23:15.315905 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:15.315907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:15.316585 (+ 678us) server_negotiation.cc:407] Connection header received
0924 00:23:15.316714 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:15.316715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:15.316727 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:15.316775 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:15.317619 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.318331 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.319359 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.319668 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.319734 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:15.321342 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:15.321344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:15.321344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:15.321353 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:15.323267 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.323969 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.323971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.323971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.324061 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.324318 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.324319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.324320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.324490 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:15.324589 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:15.324810 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 00:23:15.324898 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:23:15.326673 12711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.316330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45468 (local address 127.0.98.61:39449)
0924 00:23:15.316367 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:15.316369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:15.318651 (+ 2282us) server_negotiation.cc:407] Connection header received
0924 00:23:15.318747 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:15.318748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:15.318759 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:15.318787 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:15.319510 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.320240 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.321296 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.321640 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.321688 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:15.323199 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:15.323201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:15.323202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:15.323210 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:15.325009 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.325404 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.325406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.325406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.325466 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.325689 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.325690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.325691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.325818 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:15.326311 (+ 493us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:15.326504 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 00:23:15.326577 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:23:15.328886 12787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.316091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58286 (local address 127.0.98.60:46243)
0924 00:23:15.316130 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:15.316132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:15.316578 (+ 446us) server_negotiation.cc:407] Connection header received
0924 00:23:15.316674 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:15.316675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:15.316685 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:15.316714 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:15.317479 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.318240 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.323827 (+ 5587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.324148 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.324212 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:15.325676 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:15.325678 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:15.325678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:15.325686 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:15.327334 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.328018 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.328019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.328020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.328074 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.328326 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.328328 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.328328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.328467 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:15.328566 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:15.328712 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 00:23:15.328798 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 00:23:15.328925 12822 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:23:15.328970 12822 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:23:15.334926 12707 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } attempt: 3
Sep 24 00:23:15 dist-test-slave-777x krb5kdc[12551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209395, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 00:23:15.345865 12787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.335369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58292 (local address 127.0.98.60:46243)
0924 00:23:15.335454 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:15.335456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:15.335464 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:23:15.335485 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:15.335486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:15.335496 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:15.335548 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:15.336118 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.337011 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.338142 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.338476 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.338556 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:15.341906 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:15.341908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:15.341908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:15.341916 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:15.343879 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.344385 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.344386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.344386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.344457 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:15.345053 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:15.345054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:15.345055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:15.345409 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:15.345484 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:15.345670 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 00:23:15.345761 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 00:23:15.345865 12709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:15.335253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46243 (local address 127.0.0.1:58292)
0924 00:23:15.335339 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:15.335351 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:15.335394 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:15.335599 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:15.335600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:15.335790 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:15.335841 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:15.335843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.337118 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.337120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:15.338039 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:15.338042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:15.338562 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:15.338563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:15.338629 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:15.339172 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:15.339178 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:15.341712 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:15.343995 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:15.343996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:15.343996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:15.344292 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:15.344556 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:15.344557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:15.344557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:15.344631 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:15.345485 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:15.345486 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:15.345544 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 00:23:15.345632 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 00:23:15.345746 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 00:23:15.348028 12707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7: Bootstrap starting.
I20260924 00:23:15.348920 12707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7: Neither blocks nor log segments found. Creating new log.
I20260924 00:23:15.349261 12707 log.cc:824] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:15.350248 12707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7: No bootstrap required, opened a new log
I20260924 00:23:15.353356 12707 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f14c28cbaa43589eb74fb531aeae7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } } peers { permanent_uuid: "e9bf238337004f22b55d2272884ed4c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } } peers { permanent_uuid: "b5b8306002ce46d5b7b10f583df1bce2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } }
I20260924 00:23:15.353533 12707 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:23:15.353570 12707 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9bf238337004f22b55d2272884ed4c7, State: Initialized, Role: FOLLOWER
I20260924 00:23:15.353773 12707 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f14c28cbaa43589eb74fb531aeae7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } } peers { permanent_uuid: "e9bf238337004f22b55d2272884ed4c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } } peers { permanent_uuid: "b5b8306002ce46d5b7b10f583df1bce2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } }
I20260924 00:23:15.354183 12707 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 00:23:15.356266 12836 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:23:15.356401 12836 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:23:15.358273 12639 master_runner.cc:263] No leader master found from master e9bf238337004f22b55d2272884ed4c7
I20260924 00:23:15.358325 12639 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 00:23:15.354136 12825 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f14c28cbaa43589eb74fb531aeae7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } } peers { permanent_uuid: "e9bf238337004f22b55d2272884ed4c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } } peers { permanent_uuid: "b5b8306002ce46d5b7b10f583df1bce2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } } }
I20260924 00:23:15.358553 12825 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 00:23:15.430684 12585 tablet.cc:2369] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:23:15.519120 12735 tablet.cc:2369] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:23:15.532661 12825 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:15.532807 12825 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f14c28cbaa43589eb74fb531aeae7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } } peers { permanent_uuid: "e9bf238337004f22b55d2272884ed4c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } } peers { permanent_uuid: "b5b8306002ce46d5b7b10f583df1bce2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } }
I20260924 00:23:15.533103 12825 leader_election.cc:288] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4f14c28cbaa43589eb74fb531aeae7c (127.0.98.62:43183), b5b8306002ce46d5b7b10f583df1bce2 (127.0.98.60:46243)
I20260924 00:23:15.533385 12606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9bf238337004f22b55d2272884ed4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4f14c28cbaa43589eb74fb531aeae7c" is_pre_election: true
I20260924 00:23:15.533388 12756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9bf238337004f22b55d2272884ed4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8306002ce46d5b7b10f583df1bce2" is_pre_election: true
I20260924 00:23:15.533560 12756 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9bf238337004f22b55d2272884ed4c7 in term 0.
I20260924 00:23:15.533564 12606 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9bf238337004f22b55d2272884ed4c7 in term 0.
I20260924 00:23:15.533807 12658 leader_election.cc:302] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4f14c28cbaa43589eb74fb531aeae7c, e9bf238337004f22b55d2272884ed4c7; no voters:
I20260924 00:23:15.534053 12825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:23:15.534113 12825 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:15.534130 12825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:23:15.534879 12825 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f14c28cbaa43589eb74fb531aeae7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } } peers { permanent_uuid: "e9bf238337004f22b55d2272884ed4c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } } peers { permanent_uuid: "b5b8306002ce46d5b7b10f583df1bce2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } }
I20260924 00:23:15.535137 12825 leader_election.cc:288] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [CANDIDATE]: Term 1 election: Requested vote from peers e4f14c28cbaa43589eb74fb531aeae7c (127.0.98.62:43183), b5b8306002ce46d5b7b10f583df1bce2 (127.0.98.60:46243)
I20260924 00:23:15.535175 12756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9bf238337004f22b55d2272884ed4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5b8306002ce46d5b7b10f583df1bce2"
I20260924 00:23:15.535187 12606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9bf238337004f22b55d2272884ed4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4f14c28cbaa43589eb74fb531aeae7c"
I20260924 00:23:15.535266 12756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:23:15.535269 12606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:23:15.535934 12756 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9bf238337004f22b55d2272884ed4c7 in term 1.
I20260924 00:23:15.536139 12657 leader_election.cc:302] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5b8306002ce46d5b7b10f583df1bce2, e9bf238337004f22b55d2272884ed4c7; no voters:
I20260924 00:23:15.536242 12606 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9bf238337004f22b55d2272884ed4c7 in term 1.
I20260924 00:23:15.536255 12825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:23:15.536429 12825 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [term 1 LEADER]: Becoming Leader. State: Replica: e9bf238337004f22b55d2272884ed4c7, State: Running, Role: LEADER
I20260924 00:23:15.536532 12825 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f14c28cbaa43589eb74fb531aeae7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } } peers { permanent_uuid: "e9bf238337004f22b55d2272884ed4c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } } peers { permanent_uuid: "b5b8306002ce46d5b7b10f583df1bce2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } }
I20260924 00:23:15.536995 12838 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9bf238337004f22b55d2272884ed4c7. Latest consensus state: current_term: 1 leader_uuid: "e9bf238337004f22b55d2272884ed4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f14c28cbaa43589eb74fb531aeae7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } } peers { permanent_uuid: "e9bf238337004f22b55d2272884ed4c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } } peers { permanent_uuid: "b5b8306002ce46d5b7b10f583df1bce2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } } }
I20260924 00:23:15.537096 12838 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [sys.catalog]: This master's current role is: LEADER
I20260924 00:23:15.537303 12840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:23:15.537474 12840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:23:15.538828 12756 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2 [term 1 FOLLOWER]: Refusing update from remote peer e9bf238337004f22b55d2272884ed4c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:23:15.538872 12606 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c [term 1 FOLLOWER]: Refusing update from remote peer e9bf238337004f22b55d2272884ed4c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:23:15.539149 12838 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5b8306002ce46d5b7b10f583df1bce2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:23:15.539255 12825 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f14c28cbaa43589eb74fb531aeae7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:23:15.540436 12809 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9bf238337004f22b55d2272884ed4c7. Latest consensus state: current_term: 1 leader_uuid: "e9bf238337004f22b55d2272884ed4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f14c28cbaa43589eb74fb531aeae7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } } peers { permanent_uuid: "e9bf238337004f22b55d2272884ed4c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } } peers { permanent_uuid: "b5b8306002ce46d5b7b10f583df1bce2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } } }
I20260924 00:23:15.540557 12809 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:23:15.540522 12790 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9bf238337004f22b55d2272884ed4c7. Latest consensus state: current_term: 1 leader_uuid: "e9bf238337004f22b55d2272884ed4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f14c28cbaa43589eb74fb531aeae7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } } peers { permanent_uuid: "e9bf238337004f22b55d2272884ed4c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } } peers { permanent_uuid: "b5b8306002ce46d5b7b10f583df1bce2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } } }
I20260924 00:23:15.540608 12790 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:23:15.541460 12790 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9bf238337004f22b55d2272884ed4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f14c28cbaa43589eb74fb531aeae7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } } peers { permanent_uuid: "e9bf238337004f22b55d2272884ed4c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } } peers { permanent_uuid: "b5b8306002ce46d5b7b10f583df1bce2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } } }
I20260924 00:23:15.541450 12838 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9bf238337004f22b55d2272884ed4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f14c28cbaa43589eb74fb531aeae7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } } peers { permanent_uuid: "e9bf238337004f22b55d2272884ed4c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } } peers { permanent_uuid: "b5b8306002ce46d5b7b10f583df1bce2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } } }
I20260924 00:23:15.541551 12790 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:23:15.541550 12838 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [sys.catalog]: This master's current role is: LEADER
I20260924 00:23:15.541591 12838 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9bf238337004f22b55d2272884ed4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f14c28cbaa43589eb74fb531aeae7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } } peers { permanent_uuid: "e9bf238337004f22b55d2272884ed4c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } } peers { permanent_uuid: "b5b8306002ce46d5b7b10f583df1bce2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } } }
I20260924 00:23:15.541632 12838 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7 [sys.catalog]: This master's current role is: LEADER
I20260924 00:23:15.541694 12841 mvcc.cc:205] Tried to move back new op lower bound from 7332697684124635136 to 7332697684118188032. Current Snapshot: MvccSnapshot[applied={T|T < 7332697684124635136 or (T in {7332697684124635136})}]
I20260924 00:23:15.541813 12840 catalog_manager.cc:1347] Generated new cluster ID: f57a3f5fa0f345b29bb6b390cf465ffe
I20260924 00:23:15.541855 12840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:23:15.541896 12843 mvcc.cc:205] Tried to move back new op lower bound from 7332697684124635136 to 7332697684118188032. Current Snapshot: MvccSnapshot[applied={T|T < 7332697684124635136}]
I20260924 00:23:15.542177 12842 mvcc.cc:205] Tried to move back new op lower bound from 7332697684124635136 to 7332697684118188032. Current Snapshot: MvccSnapshot[applied={T|T < 7332697684124635136}]
I20260924 00:23:15.542250 12809 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9bf238337004f22b55d2272884ed4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4f14c28cbaa43589eb74fb531aeae7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43183 } } peers { permanent_uuid: "e9bf238337004f22b55d2272884ed4c7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39449 } } peers { permanent_uuid: "b5b8306002ce46d5b7b10f583df1bce2" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46243 } } }
I20260924 00:23:15.542310 12809 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:23:15.546236 12840 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:23:15.546911 12840 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:23:15.551255 12840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9bf238337004f22b55d2272884ed4c7: Generated new TSK 0
I20260924 00:23:15.551540 12840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:23:16.299283 12801 catalog_manager.cc:1259] Loaded cluster ID: f57a3f5fa0f345b29bb6b390cf465ffe
I20260924 00:23:16.299364 12801 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c: loading cluster ID for follower catalog manager: success
I20260924 00:23:16.300226 12801 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c: acquiring CA information for follower catalog manager: success
I20260924 00:23:16.300510 12801 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e4f14c28cbaa43589eb74fb531aeae7c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 00:23:16.329603 12822 catalog_manager.cc:1259] Loaded cluster ID: f57a3f5fa0f345b29bb6b390cf465ffe
I20260924 00:23:16.329696 12822 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2: loading cluster ID for follower catalog manager: success
I20260924 00:23:16.331130 12822 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2: acquiring CA information for follower catalog manager: success
I20260924 00:23:16.331569 12822 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b5b8306002ce46d5b7b10f583df1bce2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 00:23:16.615622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 12564
I20260924 00:23:16.621192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 12639
I20260924 00:23:16.626345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 12714
2026-09-24T00:23:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3084 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:23:16 dist-test-slave-777x krb5kdc[12855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:23:16 dist-test-slave-777x krb5kdc[12855](info): set up 2 sockets
Sep 24 00:23:16 dist-test-slave-777x krb5kdc[12855](info): commencing operation
krb5kdc: starting...
W20260924 00:23:17.711898 392 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:23:17 dist-test-slave-777x krb5kdc[12855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:23:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:23:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:23:17.840050 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43205
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:23:17.921888 12868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:17.922119 12868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:17.922142 12868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:17.923250 12868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:23:17.923286 12868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:17.923300 12868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:17.923313 12868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:23:17.923326 12868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:23:17.924767 12868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:17.924964 12868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:17.925135 12868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:17.930121 12873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:17.930186 12874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:17.930205 12868 server_base.cc:1034] running on GCE node
W20260924 00:23:17.930132 12876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:17.930612 12868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:17.930881 12868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:17.932082 12868 hybrid_clock.cc:648] HybridClock initialized: now 1790209397932049 us; error 45 us; skew 500 ppm
Sep 24 00:23:17 dist-test-slave-777x krb5kdc[12855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:17.934796 12868 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:23:17.935227 12868 webserver.cc:460] Webserver started at http://127.0.98.62:34377/ using document root <none> and password file <none>
I20260924 00:23:17.935432 12868 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:17.935478 12868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:17.935582 12868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:23:17.936798 12868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "e24bd7c3d8dd4251b4d3d51288e1af73"
format_stamp: "Formatted at 2026-09-24 00:23:17 on dist-test-slave-777x"
server_key: "fd74c52b486be1ab6b94c819acd273d9"
server_key_iv: "0e74783b20d9c467eda7ca4c4a0a073e"
server_key_version: "encryptionkey@0"
I20260924 00:23:17.937137 12868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "e24bd7c3d8dd4251b4d3d51288e1af73"
format_stamp: "Formatted at 2026-09-24 00:23:17 on dist-test-slave-777x"
server_key: "fd74c52b486be1ab6b94c819acd273d9"
server_key_iv: "0e74783b20d9c467eda7ca4c4a0a073e"
server_key_version: "encryptionkey@0"
I20260924 00:23:17.938505 12868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:23:17.939309 12883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:17.939487 12868 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:23:17.939572 12868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "e24bd7c3d8dd4251b4d3d51288e1af73"
format_stamp: "Formatted at 2026-09-24 00:23:17 on dist-test-slave-777x"
server_key: "fd74c52b486be1ab6b94c819acd273d9"
server_key_iv: "0e74783b20d9c467eda7ca4c4a0a073e"
server_key_version: "encryptionkey@0"
I20260924 00:23:17.939636 12868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:17.959190 12868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:17.961313 12868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:17.961427 12868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:17.965255 12868 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43205
I20260924 00:23:17.965306 12935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43205 every 8 connection(s)
I20260924 00:23:17.965674 12868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:23:17.966647 12936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:23:17.969204 12936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73: Bootstrap starting.
I20260924 00:23:17.969765 12936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73: Neither blocks nor log segments found. Creating new log.
I20260924 00:23:17.970022 12936 log.cc:824] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:17.970778 12936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73: No bootstrap required, opened a new log
I20260924 00:23:17.972522 12936 raft_consensus.cc:348] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e24bd7c3d8dd4251b4d3d51288e1af73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43205 } }
I20260924 00:23:17.972679 12936 raft_consensus.cc:374] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:23:17.972707 12936 raft_consensus.cc:729] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e24bd7c3d8dd4251b4d3d51288e1af73, State: Initialized, Role: FOLLOWER
I20260924 00:23:17.972827 12936 consensus_queue.cc:260] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e24bd7c3d8dd4251b4d3d51288e1af73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43205 } }
I20260924 00:23:17.972891 12936 raft_consensus.cc:388] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:23:17.972915 12936 raft_consensus.cc:482] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:23:17.972950 12936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:23:17.973539 12936 raft_consensus.cc:504] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e24bd7c3d8dd4251b4d3d51288e1af73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43205 } }
I20260924 00:23:17.973632 12936 leader_election.cc:302] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e24bd7c3d8dd4251b4d3d51288e1af73; no voters:
I20260924 00:23:17.973770 12936 leader_election.cc:288] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:23:17.973812 12939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:23:17.973961 12939 raft_consensus.cc:686] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73 [term 1 LEADER]: Becoming Leader. State: Replica: e24bd7c3d8dd4251b4d3d51288e1af73, State: Running, Role: LEADER
I20260924 00:23:17.974081 12939 consensus_queue.cc:237] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e24bd7c3d8dd4251b4d3d51288e1af73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43205 } }
I20260924 00:23:17.974118 12936 sys_catalog.cc:564] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:23:17.974462 12941 sys_catalog.cc:455] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e24bd7c3d8dd4251b4d3d51288e1af73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e24bd7c3d8dd4251b4d3d51288e1af73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43205 } } }
I20260924 00:23:17.974545 12941 sys_catalog.cc:458] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73 [sys.catalog]: This master's current role is: LEADER
I20260924 00:23:17.974592 12940 sys_catalog.cc:455] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e24bd7c3d8dd4251b4d3d51288e1af73. Latest consensus state: current_term: 1 leader_uuid: "e24bd7c3d8dd4251b4d3d51288e1af73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e24bd7c3d8dd4251b4d3d51288e1af73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43205 } } }
I20260924 00:23:17.974660 12940 sys_catalog.cc:458] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73 [sys.catalog]: This master's current role is: LEADER
I20260924 00:23:17.974802 12945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:23:17.975343 12945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:23:17.975459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 12868
I20260924 00:23:17.975561 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:23:17.975780 392 external_mini_cluster.cc:1428] Setting key d75eef016241cb8141bee23386f859f3
I20260924 00:23:17.976964 12945 catalog_manager.cc:1347] Generated new cluster ID: 8251540d457d459ebd99bc5ac112a4ff
I20260924 00:23:17.977023 12945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:23:17 dist-test-slave-777x krb5kdc[12855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:23:17.991433 12958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:17.976224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33784 (local address 127.0.98.62:43205)
0924 00:23:17.976383 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:17.976395 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:17.976431 (+ 36us) server_negotiation.cc:407] Connection header received
0924 00:23:17.976727 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:17.976735 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:17.976804 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:17.980334 (+ 3530us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:17.981189 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:17.982628 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:17.983699 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:17.984356 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:17.984457 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:17.987496 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:17.987501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:17.987502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:17.987517 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:17.989680 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:17.990080 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:17.990084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:17.990084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:17.990226 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:17.990434 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:17.990435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:17.990436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:17.990776 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:17.990922 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:17.991023 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 00:23:17.991175 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260924 00:23:17.992566 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:23:17.995914 12945 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:23:17.996676 12945 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:23:18.004204 12945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e24bd7c3d8dd4251b4d3d51288e1af73: Generated new TSK 0
I20260924 00:23:18.004434 12945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:23:18.046924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 12868
2026-09-24T00:23:18Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1419 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:23:18 dist-test-slave-777x krb5kdc[12963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:23:18 dist-test-slave-777x krb5kdc[12963](info): set up 2 sockets
Sep 24 00:23:18 dist-test-slave-777x krb5kdc[12963](info): commencing operation
krb5kdc: starting...
W20260924 00:23:19.192400 392 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:23:19 dist-test-slave-777x krb5kdc[12963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:23:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:23:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:23:19.324980 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36285
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:23:19.411418 12976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:19.411662 12976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:19.411691 12976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:19.413095 12976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:23:19.413146 12976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:19.413162 12976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:19.413177 12976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:23:19.413190 12976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:23:19.414860 12976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:19.415138 12976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:19.415398 12976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:19.420535 12981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:19.420535 12984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:19.420639 12982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:19.420819 12976 server_base.cc:1034] running on GCE node
I20260924 00:23:19.421175 12976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:19.421476 12976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:19.422664 12976 hybrid_clock.cc:648] HybridClock initialized: now 1790209399422643 us; error 33 us; skew 500 ppm
Sep 24 00:23:19 dist-test-slave-777x krb5kdc[12963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:19.429612 12976 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:23:19.430151 12976 webserver.cc:460] Webserver started at http://127.0.98.62:44051/ using document root <none> and password file <none>
I20260924 00:23:19.430402 12976 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:19.430480 12976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:19.430634 12976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:23:19.431751 12976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "5433edd2f6ce47dcb09947233e09fd0c"
format_stamp: "Formatted at 2026-09-24 00:23:19 on dist-test-slave-777x"
server_key: "85bc76a6fe59a976bdb1339969d3ac83"
server_key_iv: "aac536a41a9230797a9399f586d9558e"
server_key_version: "encryptionkey@0"
I20260924 00:23:19.432155 12976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "5433edd2f6ce47dcb09947233e09fd0c"
format_stamp: "Formatted at 2026-09-24 00:23:19 on dist-test-slave-777x"
server_key: "85bc76a6fe59a976bdb1339969d3ac83"
server_key_iv: "aac536a41a9230797a9399f586d9558e"
server_key_version: "encryptionkey@0"
I20260924 00:23:19.433670 12976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:23:19.434520 12991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:19.434692 12976 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:23:19.434791 12976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "5433edd2f6ce47dcb09947233e09fd0c"
format_stamp: "Formatted at 2026-09-24 00:23:19 on dist-test-slave-777x"
server_key: "85bc76a6fe59a976bdb1339969d3ac83"
server_key_iv: "aac536a41a9230797a9399f586d9558e"
server_key_version: "encryptionkey@0"
I20260924 00:23:19.434873 12976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:19.459738 12976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:19.462766 12976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:19.462996 12976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:19.467880 12976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36285
I20260924 00:23:19.467917 13043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36285 every 8 connection(s)
I20260924 00:23:19.468382 12976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:23:19.469566 13044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:23:19.470686 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 12976
I20260924 00:23:19.470810 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:23:19.471043 392 external_mini_cluster.cc:1428] Setting key af965c8cd473835c979b19b343f986a9
I20260924 00:23:19.473451 13044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c: Bootstrap starting.
I20260924 00:23:19.474125 13044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c: Neither blocks nor log segments found. Creating new log.
I20260924 00:23:19.474440 13044 log.cc:824] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:19.475189 13044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c: No bootstrap required, opened a new log
I20260924 00:23:19.477288 13044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5433edd2f6ce47dcb09947233e09fd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } }
I20260924 00:23:19.477463 13044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:23:19.477505 13044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5433edd2f6ce47dcb09947233e09fd0c, State: Initialized, Role: FOLLOWER
I20260924 00:23:19.477665 13044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5433edd2f6ce47dcb09947233e09fd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } }
I20260924 00:23:19.477754 13044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:23:19.477820 13044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:23:19.477900 13044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 0 FOLLOWER]: Advancing to term 1
Sep 24 00:23:19 dist-test-slave-777x krb5kdc[12963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:23:19.478600 13044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5433edd2f6ce47dcb09947233e09fd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } }
I20260924 00:23:19.478725 13044 leader_election.cc:302] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5433edd2f6ce47dcb09947233e09fd0c; no voters:
I20260924 00:23:19.478943 13044 leader_election.cc:288] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:23:19.478982 13049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:23:19.479198 13049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 1 LEADER]: Becoming Leader. State: Replica: 5433edd2f6ce47dcb09947233e09fd0c, State: Running, Role: LEADER
I20260924 00:23:19.479343 13049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5433edd2f6ce47dcb09947233e09fd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } }
I20260924 00:23:19.479373 13044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:23:19.481900 13050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5433edd2f6ce47dcb09947233e09fd0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5433edd2f6ce47dcb09947233e09fd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } } }
I20260924 00:23:19.481923 13051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5433edd2f6ce47dcb09947233e09fd0c. Latest consensus state: current_term: 1 leader_uuid: "5433edd2f6ce47dcb09947233e09fd0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5433edd2f6ce47dcb09947233e09fd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } } }
I20260924 00:23:19.482092 13051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [sys.catalog]: This master's current role is: LEADER
I20260924 00:23:19.482504 13050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [sys.catalog]: This master's current role is: LEADER
I20260924 00:23:19.483290 13047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:19.471627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59464 (local address 127.0.98.62:36285)
0924 00:23:19.471803 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:19.471811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:19.471836 (+ 25us) server_negotiation.cc:407] Connection header received
0924 00:23:19.472326 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:19.472335 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:19.472387 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:19.472506 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:19.473396 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:19.474661 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:19.475552 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:19.476013 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:19.476153 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:19.478938 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:19.478943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:19.478943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:19.478963 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:19.481028 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:19.481507 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:19.481526 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:19.481527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:19.481662 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:19.482057 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:19.482058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:19.482058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:19.482454 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:19.482626 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:19.482990 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 00:23:19.483138 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
W20260924 00:23:19.482789 13063 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:23:19.483410 13063 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:23:19.483479 13064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:23:19.483670 13064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:23:19.485324 13064 catalog_manager.cc:1347] Generated new cluster ID: a2e4f7ae143946d194025989511b750c
I20260924 00:23:19.485386 13064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:23:19.507364 13064 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:23:19.507974 13064 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:23:19.512799 13064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c: Generated new TSK 0
I20260924 00:23:19.513042 13064 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:23:19.548811 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36285
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:23:19.636858 13070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:19.637086 13070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:19.637113 13070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:19.638321 13070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:19.638368 13070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:19.638396 13070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:23:19.639947 13070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:19.640247 13070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:19.640504 13070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:19.645707 13078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:19.645707 13075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:19.645818 13076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:19.646175 13070 server_base.cc:1034] running on GCE node
I20260924 00:23:19.646368 13070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:19.646613 13070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:19.647794 13070 hybrid_clock.cc:648] HybridClock initialized: now 1790209399647779 us; error 29 us; skew 500 ppm
Sep 24 00:23:19 dist-test-slave-777x krb5kdc[12963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:19.650411 13070 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:23:19.650906 13070 webserver.cc:460] Webserver started at http://127.0.98.1:37471/ using document root <none> and password file <none>
I20260924 00:23:19.651153 13070 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:19.651218 13070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:19.651357 13070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:23:19.652580 13070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "c68a46d873bb4eb39681360c4e010e18"
format_stamp: "Formatted at 2026-09-24 00:23:19 on dist-test-slave-777x"
server_key: "2efeee16bceb925c8602d986365d9fb1"
server_key_iv: "7ab21b57d3f0f3ff5932d160fec85dd3"
server_key_version: "encryptionkey@0"
I20260924 00:23:19.652956 13070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c68a46d873bb4eb39681360c4e010e18"
format_stamp: "Formatted at 2026-09-24 00:23:19 on dist-test-slave-777x"
server_key: "2efeee16bceb925c8602d986365d9fb1"
server_key_iv: "7ab21b57d3f0f3ff5932d160fec85dd3"
server_key_version: "encryptionkey@0"
I20260924 00:23:19.654448 13070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:23:19.655403 13085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:19.655625 13070 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:23:19.655735 13070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "c68a46d873bb4eb39681360c4e010e18"
format_stamp: "Formatted at 2026-09-24 00:23:19 on dist-test-slave-777x"
server_key: "2efeee16bceb925c8602d986365d9fb1"
server_key_iv: "7ab21b57d3f0f3ff5932d160fec85dd3"
server_key_version: "encryptionkey@0"
I20260924 00:23:19.655810 13070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:19.671205 13070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:19.673432 13070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:19.673599 13070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:19.673837 13070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:23:19.674180 13070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:23:19.674229 13070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:19.674254 13070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:23:19.674270 13070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:19.680421 13070 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45067
I20260924 00:23:19.680495 13198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45067 every 8 connection(s)
I20260924 00:23:19.680810 13070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:23:19.684206 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 13070
I20260924 00:23:19.684319 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:23:19.684549 392 external_mini_cluster.cc:1428] Setting key 04d4c43c96c1b876ac28f3ac1c77b59b
Sep 24 00:23:19 dist-test-slave-777x krb5kdc[12963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:23:19.691913 13047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:19.681679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44315 (local address 127.0.98.62:36285)
0924 00:23:19.681727 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:19.681729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:19.681904 (+ 175us) server_negotiation.cc:407] Connection header received
0924 00:23:19.682387 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:19.682388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:19.682399 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:19.682480 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:19.683597 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:19.684346 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:19.685631 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:19.685979 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:19.686057 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:19.688738 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:19.688740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:19.688740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:19.688748 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:19.690402 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:19.690958 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:19.690959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:19.690960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:19.691020 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:19.691264 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:19.691265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:19.691265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:19.691383 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:19.691455 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:19.691726 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 00:23:19.691820 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:23:19.692137 13201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:19.681593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.1:44315)
0924 00:23:19.681755 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:19.681797 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:19.682278 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:19.682586 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:19.682590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:19.683260 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:19.683476 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:19.683481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:19.684446 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:19.684451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:19.685549 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:19.685552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:19.686103 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:19.686104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:19.686233 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:19.686868 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:19.686881 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:19.688638 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:19.690618 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:19.690627 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:19.690630 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:19.690868 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:19.691086 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:19.691087 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:19.691087 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:19.691198 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:19.691446 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:19.691453 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:19.691599 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 00:23:19.691708 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 00:23:19.691875 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260924 00:23:19.692713 13199 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36285
I20260924 00:23:19.692840 13199 heartbeater.cc:464] Registering TS with master...
I20260924 00:23:19.693100 13199 heartbeater.cc:511] Master 127.0.98.62:36285 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 00:23:19.693719 13008 ts_manager.cc:194] Registered new tserver with Master: c68a46d873bb4eb39681360c4e010e18 (127.0.98.1:45067)
I20260924 00:23:19.694355 13008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:23:19.715927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36285
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:23:19.798182 13204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:19.798434 13204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:19.798461 13204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:19.799670 13204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:19.799715 13204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:19.799743 13204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:23:19.801301 13204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:19.801571 13204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:19.801815 13204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:19.807154 13210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:19.807160 13212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:19.807271 13204 server_base.cc:1034] running on GCE node
W20260924 00:23:19.807179 13209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:19.807682 13204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:19.807894 13204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:19.809062 13204 hybrid_clock.cc:648] HybridClock initialized: now 1790209399809037 us; error 37 us; skew 500 ppm
Sep 24 00:23:19 dist-test-slave-777x krb5kdc[12963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:19.811877 13204 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:23:19.812579 13204 webserver.cc:460] Webserver started at http://127.0.98.2:41759/ using document root <none> and password file <none>
I20260924 00:23:19.812824 13204 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:19.812891 13204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:19.813046 13204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:23:19.814168 13204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "c702a480526c43919e56ce508cab9167"
format_stamp: "Formatted at 2026-09-24 00:23:19 on dist-test-slave-777x"
server_key: "7105cb485a44ae200b7f5c19ca750aa7"
server_key_iv: "585f9f41b930a0c52bbbe87c4ee0075f"
server_key_version: "encryptionkey@0"
I20260924 00:23:19.814555 13204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c702a480526c43919e56ce508cab9167"
format_stamp: "Formatted at 2026-09-24 00:23:19 on dist-test-slave-777x"
server_key: "7105cb485a44ae200b7f5c19ca750aa7"
server_key_iv: "585f9f41b930a0c52bbbe87c4ee0075f"
server_key_version: "encryptionkey@0"
I20260924 00:23:19.816134 13204 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:23:19.817124 13219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:19.817328 13204 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:23:19.817417 13204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "c702a480526c43919e56ce508cab9167"
format_stamp: "Formatted at 2026-09-24 00:23:19 on dist-test-slave-777x"
server_key: "7105cb485a44ae200b7f5c19ca750aa7"
server_key_iv: "585f9f41b930a0c52bbbe87c4ee0075f"
server_key_version: "encryptionkey@0"
I20260924 00:23:19.817480 13204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:19.834862 13204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:19.837294 13204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:19.837411 13204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:19.837648 13204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:23:19.837975 13204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:23:19.838009 13204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:19.838059 13204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:23:19.838089 13204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:19.843626 13204 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39327
I20260924 00:23:19.843703 13332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39327 every 8 connection(s)
I20260924 00:23:19.844017 13204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 00:23:19 dist-test-slave-777x krb5kdc[12963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:23:19.851256 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 13204
I20260924 00:23:19.851379 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:23:19.851634 392 external_mini_cluster.cc:1428] Setting key 5b2fe162706e840a21557633e05f208d
I20260924 00:23:19.853709 13047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:19.844806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37267 (local address 127.0.98.62:36285)
0924 00:23:19.844878 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:19.844880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:19.845101 (+ 221us) server_negotiation.cc:407] Connection header received
0924 00:23:19.845591 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:19.845592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:19.845601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:19.845655 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:19.846597 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:19.847093 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:19.848105 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:19.848366 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:19.848433 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:19.850481 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:19.850483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:19.850483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:19.850490 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:19.852224 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:19.852707 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:19.852709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:19.852709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:19.852768 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:19.853064 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:19.853064 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:19.853065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:19.853168 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:19.853247 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:19.853524 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 00:23:19.853619 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 00:23:19.853904 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:19.844789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:37267)
0924 00:23:19.844957 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:19.845003 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:19.845508 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:19.845722 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:19.845725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:19.846305 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:19.846521 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:19.846526 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:19.847177 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:19.847178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:19.847996 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:19.848004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:19.848439 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:19.848440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:19.848552 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:19.849033 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:19.849043 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:19.850399 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:19.852365 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:19.852371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:19.852373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:19.852628 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:19.852823 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:19.852824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:19.852824 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:19.852936 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:19.853239 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:19.853246 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:19.853378 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 00:23:19.853500 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 00:23:19.853658 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
I20260924 00:23:19.854331 13333 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36285
I20260924 00:23:19.854440 13333 heartbeater.cc:464] Registering TS with master...
I20260924 00:23:19.854620 13333 heartbeater.cc:511] Master 127.0.98.62:36285 requested a full tablet report, sending...
I20260924 00:23:19.855023 13008 ts_manager.cc:194] Registered new tserver with Master: c702a480526c43919e56ce508cab9167 (127.0.98.2:39327)
I20260924 00:23:19.855392 13008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37267
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:23:19.879722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36285
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:23:19.966395 13338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:19.966635 13338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:19.966665 13338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:19.967976 13338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:19.968022 13338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:19.968051 13338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:23:19.969607 13338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:19.969870 13338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:19.970115 13338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:19.975420 13346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:19.975456 13344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:19.975757 13338 server_base.cc:1034] running on GCE node
W20260924 00:23:19.975837 13343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:19.976104 13338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:19.976373 13338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:19.977548 13338 hybrid_clock.cc:648] HybridClock initialized: now 1790209399977533 us; error 32 us; skew 500 ppm
Sep 24 00:23:19 dist-test-slave-777x krb5kdc[12963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:19.980422 13338 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:23:19.980940 13338 webserver.cc:460] Webserver started at http://127.0.98.3:39053/ using document root <none> and password file <none>
I20260924 00:23:19.981169 13338 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:19.981232 13338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:19.981374 13338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:23:19.982764 13338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "b827fb68e9a44201a364856308b8432b"
format_stamp: "Formatted at 2026-09-24 00:23:19 on dist-test-slave-777x"
server_key: "8f9f1528f2063b5d3210f4374d0f189c"
server_key_iv: "9cc0d81a33b052d3fe0d6946317a60f9"
server_key_version: "encryptionkey@0"
I20260924 00:23:19.983148 13338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b827fb68e9a44201a364856308b8432b"
format_stamp: "Formatted at 2026-09-24 00:23:19 on dist-test-slave-777x"
server_key: "8f9f1528f2063b5d3210f4374d0f189c"
server_key_iv: "9cc0d81a33b052d3fe0d6946317a60f9"
server_key_version: "encryptionkey@0"
I20260924 00:23:19.984783 13338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260924 00:23:19.985620 13353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:19.985908 13338 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:23:19.986037 13338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "b827fb68e9a44201a364856308b8432b"
format_stamp: "Formatted at 2026-09-24 00:23:19 on dist-test-slave-777x"
server_key: "8f9f1528f2063b5d3210f4374d0f189c"
server_key_iv: "9cc0d81a33b052d3fe0d6946317a60f9"
server_key_version: "encryptionkey@0"
I20260924 00:23:19.986114 13338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:20.003963 13338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:20.006352 13338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:20.006529 13338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:20.006785 13338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:23:20.007100 13338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:23:20.007153 13338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:20.007193 13338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:23:20.007225 13338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:20.013587 13338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37021
I20260924 00:23:20.013652 13466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37021 every 8 connection(s)
I20260924 00:23:20.013979 13338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:23:20.015339 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 13338
I20260924 00:23:20.015447 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:23:20.015659 392 external_mini_cluster.cc:1428] Setting key a5b53f02d82c1177183ade1d672532b6
Sep 24 00:23:20 dist-test-slave-777x krb5kdc[12963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:23:20.024457 13047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.014792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38951 (local address 127.0.98.62:36285)
0924 00:23:20.014859 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:20.014861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:20.015105 (+ 244us) server_negotiation.cc:407] Connection header received
0924 00:23:20.015736 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:20.015738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:20.015750 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:20.015803 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:20.016876 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.017410 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.018436 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.018638 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.018721 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.021244 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:20.021246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:20.021246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:20.021255 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:20.022910 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:20.023370 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:20.023371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:20.023372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:20.023433 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:20.023732 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:20.023733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:20.023734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:20.023858 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:20.023931 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:20.024273 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 00:23:20.024360 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 00:23:20.024662 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.014792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:38951)
0924 00:23:20.014962 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:20.015000 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:20.015627 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:20.015892 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:20.015896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:20.016575 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:20.016776 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.016781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.017509 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.017510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.018328 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.018330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.018729 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.018730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.018854 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.019452 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:20.019462 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:20.021134 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:20.023029 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:20.023037 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:20.023040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:20.023272 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:20.023509 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:20.023510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:20.023510 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:20.023654 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:20.023920 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:20.023927 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:20.024095 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 00:23:20.024245 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 00:23:20.024411 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260924 00:23:20.025156 13467 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36285
I20260924 00:23:20.025264 13467 heartbeater.cc:464] Registering TS with master...
I20260924 00:23:20.025444 13467 heartbeater.cc:511] Master 127.0.98.62:36285 requested a full tablet report, sending...
I20260924 00:23:20.025804 13008 ts_manager.cc:194] Registered new tserver with Master: b827fb68e9a44201a364856308b8432b (127.0.98.3:37021)
I20260924 00:23:20.026216 13008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38951
I20260924 00:23:20.027370 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:23:20.034996 13047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.028721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59466 (local address 127.0.98.62:36285)
0924 00:23:20.028820 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:20.028822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:20.028930 (+ 108us) server_negotiation.cc:407] Connection header received
0924 00:23:20.029081 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:20.029082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:20.029092 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:20.029138 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:20.029664 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.030163 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.030894 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.031097 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.031176 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.032423 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:20.032424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:20.032425 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:20.032432 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:20.033722 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:20.034135 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:20.034136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:20.034137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:20.034202 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:20.034408 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:20.034409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:20.034409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:20.034577 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:20.034691 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:20.034806 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 00:23:20.034894 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 00:23:20.036669 13008 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:23:20.038415 13008 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:23:20.045634 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.041440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40254 (local address 127.0.98.1:45067)
0924 00:23:20.041627 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:20.041632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:20.042076 (+ 444us) server_negotiation.cc:407] Connection header received
0924 00:23:20.042121 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:20.042124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:20.042140 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:20.042211 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:23:20.042396 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.043098 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.044554 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.045193 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.045276 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.045343 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:20.045493 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 00:23:20.045537 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
I20260924 00:23:20.045624 13479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.041322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45067 (local address 127.0.0.1:40254)
0924 00:23:20.041968 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:20.041977 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:20.042019 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:20.042290 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:20.042291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:20.042294 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:23:20.042332 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.042334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.043195 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.043196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.044455 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.044458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.045283 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.045284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.045369 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.045386 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 00:23:20.045462 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 00:23:20.045522 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":64,"threads_started":1}
I20260924 00:23:20.045804 13478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.041091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39327 (local address 127.0.0.1:50462)
0924 00:23:20.041440 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:20.041498 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:20.041577 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:20.041928 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:20.041931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:20.041934 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:23:20.042051 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.042053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.043513 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.043514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.044480 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.044482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.045458 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.045459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.045540 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.045543 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:23:20.045591 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 00:23:20.045625 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":1,"thread_start_us":81,"threads_started":1}
I20260924 00:23:20.046090 13480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.041237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50462 (local address 127.0.98.2:39327)
0924 00:23:20.041652 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:20.041657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:20.041670 (+ 13us) server_negotiation.cc:407] Connection header received
0924 00:23:20.041716 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:20.041718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:20.041749 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:20.041830 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:23:20.042145 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.043387 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.044583 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.045307 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.045371 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.045425 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:20.045665 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 00:23:20.046006 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":97,"threads_started":1}
I20260924 00:23:20.046605 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.041108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37021 (local address 127.0.0.1:48876)
0924 00:23:20.041794 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:20.041804 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:20.041876 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:20.042075 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:20.042076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:20.042077 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:23:20.042135 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.042137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.043885 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.043886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.044979 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.044982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.046361 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.046362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.046427 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.046431 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:23:20.046498 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 00:23:20.046536 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":76,"threads_started":1}
I20260924 00:23:20.046643 13133 tablet_service.cc:1467] Processing CreateTablet for tablet c7f3d95e9180443abecaac1c92f43ab8 (DEFAULT_TABLE table=test-table [id=837fb37c92a64edeb98c6ad7ebde94a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:23:20.046639 13483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.041448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48876 (local address 127.0.98.3:37021)
0924 00:23:20.041745 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:20.041749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:20.041874 (+ 125us) server_negotiation.cc:407] Connection header received
0924 00:23:20.041945 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:20.041947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:20.041964 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:20.042076 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:23:20.042360 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.043528 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.045078 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.046274 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.046353 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.046406 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:20.046508 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 00:23:20.046550 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":118,"threads_started":1}
I20260924 00:23:20.046809 13267 tablet_service.cc:1467] Processing CreateTablet for tablet c7f3d95e9180443abecaac1c92f43ab8 (DEFAULT_TABLE table=test-table [id=837fb37c92a64edeb98c6ad7ebde94a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:23:20.047065 13133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7f3d95e9180443abecaac1c92f43ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:23:20.047175 13267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7f3d95e9180443abecaac1c92f43ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:23:20.049289 13401 tablet_service.cc:1467] Processing CreateTablet for tablet c7f3d95e9180443abecaac1c92f43ab8 (DEFAULT_TABLE table=test-table [id=837fb37c92a64edeb98c6ad7ebde94a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:23:20.049625 13401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7f3d95e9180443abecaac1c92f43ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:23:20.050513 13485 tablet_bootstrap.cc:492] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18: Bootstrap starting.
I20260924 00:23:20.050719 13484 tablet_bootstrap.cc:492] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167: Bootstrap starting.
I20260924 00:23:20.051187 13485 tablet_bootstrap.cc:654] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18: Neither blocks nor log segments found. Creating new log.
I20260924 00:23:20.051257 13484 tablet_bootstrap.cc:654] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167: Neither blocks nor log segments found. Creating new log.
I20260924 00:23:20.051767 13484 log.cc:824] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:20.051765 13485 log.cc:824] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:20.052506 13484 tablet_bootstrap.cc:492] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167: No bootstrap required, opened a new log
I20260924 00:23:20.052554 13484 ts_tablet_manager.cc:1397] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:23:20.052893 13488 tablet_bootstrap.cc:492] T c7f3d95e9180443abecaac1c92f43ab8 P b827fb68e9a44201a364856308b8432b: Bootstrap starting.
I20260924 00:23:20.053511 13488 tablet_bootstrap.cc:654] T c7f3d95e9180443abecaac1c92f43ab8 P b827fb68e9a44201a364856308b8432b: Neither blocks nor log segments found. Creating new log.
I20260924 00:23:20.053707 13485 tablet_bootstrap.cc:492] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18: No bootstrap required, opened a new log
I20260924 00:23:20.053787 13485 ts_tablet_manager.cc:1397] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:23:20.054054 13488 log.cc:824] T c7f3d95e9180443abecaac1c92f43ab8 P b827fb68e9a44201a364856308b8432b: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:20.054729 13488 tablet_bootstrap.cc:492] T c7f3d95e9180443abecaac1c92f43ab8 P b827fb68e9a44201a364856308b8432b: No bootstrap required, opened a new log
I20260924 00:23:20.054805 13488 ts_tablet_manager.cc:1397] T c7f3d95e9180443abecaac1c92f43ab8 P b827fb68e9a44201a364856308b8432b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:23:20.054778 13484 raft_consensus.cc:348] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:20.054994 13484 raft_consensus.cc:374] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:23:20.055050 13484 raft_consensus.cc:729] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c702a480526c43919e56ce508cab9167, State: Initialized, Role: FOLLOWER
I20260924 00:23:20.055173 13484 consensus_queue.cc:260] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:20.055471 13484 ts_tablet_manager.cc:1428] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:23:20.055490 13333 heartbeater.cc:503] Master 127.0.98.62:36285 was elected leader, sending a full tablet report...
I20260924 00:23:20.056001 13485 raft_consensus.cc:348] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:20.056175 13485 raft_consensus.cc:374] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:23:20.056268 13485 raft_consensus.cc:729] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68a46d873bb4eb39681360c4e010e18, State: Initialized, Role: FOLLOWER
I20260924 00:23:20.056399 13485 consensus_queue.cc:260] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:20.056704 13485 ts_tablet_manager.cc:1428] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 00:23:20.056840 13199 heartbeater.cc:503] Master 127.0.98.62:36285 was elected leader, sending a full tablet report...
I20260924 00:23:20.056978 13488 raft_consensus.cc:348] T c7f3d95e9180443abecaac1c92f43ab8 P b827fb68e9a44201a364856308b8432b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:20.057142 13488 raft_consensus.cc:374] T c7f3d95e9180443abecaac1c92f43ab8 P b827fb68e9a44201a364856308b8432b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:23:20.057188 13488 raft_consensus.cc:729] T c7f3d95e9180443abecaac1c92f43ab8 P b827fb68e9a44201a364856308b8432b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b827fb68e9a44201a364856308b8432b, State: Initialized, Role: FOLLOWER
I20260924 00:23:20.057335 13488 consensus_queue.cc:260] T c7f3d95e9180443abecaac1c92f43ab8 P b827fb68e9a44201a364856308b8432b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:20.057596 13488 ts_tablet_manager.cc:1428] T c7f3d95e9180443abecaac1c92f43ab8 P b827fb68e9a44201a364856308b8432b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:23:20.057711 13467 heartbeater.cc:503] Master 127.0.98.62:36285 was elected leader, sending a full tablet report...
I20260924 00:23:20.092540 13490 raft_consensus.cc:482] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:20.092674 13490 raft_consensus.cc:504] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:20.093093 13490 leader_election.cc:288] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b827fb68e9a44201a364856308b8432b (127.0.98.3:37021), c68a46d873bb4eb39681360c4e010e18 (127.0.98.1:45067)
W20260924 00:23:20.094614 13334 tablet.cc:2369] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:23:20.096503 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.093364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37021 (local address 127.0.98.2:58659)
0924 00:23:20.093430 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:20.093445 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:20.093512 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:20.093678 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:20.093679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:20.093682 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:23:20.093764 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.093766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.094667 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.094668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.095600 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.095602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.096208 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.096209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.096301 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.096307 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 00:23:20.096378 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 00:23:20.096428 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 00:23:20.096545 13483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.093432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58659 (local address 127.0.98.3:37021)
0924 00:23:20.093504 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:20.093506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:20.093519 (+ 13us) server_negotiation.cc:407] Connection header received
0924 00:23:20.093558 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:20.093559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:20.093573 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:20.093654 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:23:20.093843 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.094570 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.095701 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.096110 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.096186 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.096218 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:20.096408 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 00:23:20.096453 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 00:23:20.096719 13493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.093495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45067 (local address 127.0.98.2:55171)
0924 00:23:20.093696 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:20.093707 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:20.093750 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:20.093899 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:20.093900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:20.093901 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:23:20.094038 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.094040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.094928 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.094929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.095679 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.095681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.096412 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.096413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.096496 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.096500 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:23:20.096571 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 00:23:20.096614 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":125,"threads_started":1}
I20260924 00:23:20.096726 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.093686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55171 (local address 127.0.98.1:45067)
0924 00:23:20.093743 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:20.093745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:20.093755 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:23:20.093814 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:20.093815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:20.093830 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:20.093882 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:23:20.094158 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.094850 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.095774 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.096302 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.096387 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.096397 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:20.096586 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 00:23:20.096639 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 00:23:20.096926 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f3d95e9180443abecaac1c92f43ab8" candidate_uuid: "c702a480526c43919e56ce508cab9167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b827fb68e9a44201a364856308b8432b" is_pre_election: true
I20260924 00:23:20.097131 13421 raft_consensus.cc:2454] T c7f3d95e9180443abecaac1c92f43ab8 P b827fb68e9a44201a364856308b8432b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c702a480526c43919e56ce508cab9167 in term 0.
I20260924 00:23:20.097416 13221 leader_election.cc:302] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b827fb68e9a44201a364856308b8432b, c702a480526c43919e56ce508cab9167; no voters:
I20260924 00:23:20.097540 13153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f3d95e9180443abecaac1c92f43ab8" candidate_uuid: "c702a480526c43919e56ce508cab9167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c68a46d873bb4eb39681360c4e010e18" is_pre_election: true
I20260924 00:23:20.097589 13490 raft_consensus.cc:2781] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:23:20.097659 13490 raft_consensus.cc:482] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:20.097703 13490 raft_consensus.cc:3037] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:23:20.097718 13153 raft_consensus.cc:2454] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c702a480526c43919e56ce508cab9167 in term 0.
I20260924 00:23:20.098537 13490 raft_consensus.cc:504] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:20.098676 13490 leader_election.cc:288] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167 [CANDIDATE]: Term 1 election: Requested vote from peers b827fb68e9a44201a364856308b8432b (127.0.98.3:37021), c68a46d873bb4eb39681360c4e010e18 (127.0.98.1:45067)
I20260924 00:23:20.098856 13153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f3d95e9180443abecaac1c92f43ab8" candidate_uuid: "c702a480526c43919e56ce508cab9167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c68a46d873bb4eb39681360c4e010e18"
I20260924 00:23:20.098843 13421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7f3d95e9180443abecaac1c92f43ab8" candidate_uuid: "c702a480526c43919e56ce508cab9167" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b827fb68e9a44201a364856308b8432b"
I20260924 00:23:20.098937 13421 raft_consensus.cc:3037] T c7f3d95e9180443abecaac1c92f43ab8 P b827fb68e9a44201a364856308b8432b [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:23:20.098937 13153 raft_consensus.cc:3037] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:23:20.099920 13153 raft_consensus.cc:2454] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c702a480526c43919e56ce508cab9167 in term 1.
I20260924 00:23:20.099920 13421 raft_consensus.cc:2454] T c7f3d95e9180443abecaac1c92f43ab8 P b827fb68e9a44201a364856308b8432b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c702a480526c43919e56ce508cab9167 in term 1.
I20260924 00:23:20.100133 13221 leader_election.cc:302] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c68a46d873bb4eb39681360c4e010e18, c702a480526c43919e56ce508cab9167; no voters:
I20260924 00:23:20.100275 13490 raft_consensus.cc:2781] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:23:20.100442 13490 raft_consensus.cc:686] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167 [term 1 LEADER]: Becoming Leader. State: Replica: c702a480526c43919e56ce508cab9167, State: Running, Role: LEADER
I20260924 00:23:20.100550 13490 consensus_queue.cc:237] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:20.101416 13007 catalog_manager.cc:5495] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167 reported cstate change: term changed from 0 to 1, leader changed from <none> to c702a480526c43919e56ce508cab9167 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c702a480526c43919e56ce508cab9167" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20260924 00:23:20.119266 13480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.116506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50474 (local address 127.0.98.2:39327)
0924 00:23:20.116584 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:20.116587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:20.116598 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:23:20.116669 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:20.116670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:20.116699 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:20.116783 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:23:20.117010 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.117521 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.118382 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.118601 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.118681 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.118868 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:23:20.118993 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:23:20.119094 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:20.119115 (+ 21us) server_negotiation.cc:299] Negotiation successful
0924 00:23:20.119137 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 00:23:20.121467 13421 raft_consensus.cc:1261] T c7f3d95e9180443abecaac1c92f43ab8 P b827fb68e9a44201a364856308b8432b [term 1 FOLLOWER]: Refusing update from remote peer c702a480526c43919e56ce508cab9167: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:23:20.121467 13153 raft_consensus.cc:1261] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18 [term 1 FOLLOWER]: Refusing update from remote peer c702a480526c43919e56ce508cab9167: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:23:20.121842 13490 consensus_queue.cc:1035] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:23:20.122056 13494 consensus_queue.cc:1035] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:23:20.124457 13497 mvcc.cc:205] Tried to move back new op lower bound from 7332697702894456832 to 7332697702812409856. Current Snapshot: MvccSnapshot[applied={T|T < 7332697702894456832}]
I20260924 00:23:20.125146 13498 mvcc.cc:205] Tried to move back new op lower bound from 7332697702894456832 to 7332697702812409856. Current Snapshot: MvccSnapshot[applied={T|T < 7332697702894456832}]
I20260924 00:23:20.125478 13499 mvcc.cc:205] Tried to move back new op lower bound from 7332697702894456832 to 7332697702812409856. Current Snapshot: MvccSnapshot[applied={T|T < 7332697702894456832}]
I20260924 00:23:20.127079 13007 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:23:20.127192 13007 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:23:20.128000 13007 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c: Sending DeleteTablet for 3 replicas of tablet c7f3d95e9180443abecaac1c92f43ab8
I20260924 00:23:20.128400 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 13070
I20260924 00:23:20.128391 13133 tablet_service.cc:1514] Processing DeleteTablet for tablet c7f3d95e9180443abecaac1c92f43ab8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:23:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40254
I20260924 00:23:20.128371 13267 tablet_service.cc:1514] Processing DeleteTablet for tablet c7f3d95e9180443abecaac1c92f43ab8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:23:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50462
I20260924 00:23:20.128374 13401 tablet_service.cc:1514] Processing DeleteTablet for tablet c7f3d95e9180443abecaac1c92f43ab8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:23:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48876
I20260924 00:23:20.129122 13506 tablet_replica.cc:330] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167: stopping tablet replica
I20260924 00:23:20.129228 13506 raft_consensus.cc:2229] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:23:20.129323 13506 raft_consensus.cc:2258] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:23:20.129719 13506 ts_tablet_manager.cc:1905] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:23:20.131659 13506 ts_tablet_manager.cc:1918] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:23:20.131724 13506 log.cc:1196] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/wal/wals/c7f3d95e9180443abecaac1c92f43ab8
I20260924 00:23:20.131989 13506 ts_tablet_manager.cc:1939] T c7f3d95e9180443abecaac1c92f43ab8 P c702a480526c43919e56ce508cab9167: Deleting consensus metadata
I20260924 00:23:20.132426 12992 catalog_manager.cc:4841] TS c702a480526c43919e56ce508cab9167 (127.0.98.2:39327): tablet c7f3d95e9180443abecaac1c92f43ab8 (table test-table [id=837fb37c92a64edeb98c6ad7ebde94a2]) successfully deleted
W20260924 00:23:20.134370 12993 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45067 (error 108)
I20260924 00:23:20.134527 13507 tablet_replica.cc:330] T c7f3d95e9180443abecaac1c92f43ab8 P b827fb68e9a44201a364856308b8432b: stopping tablet replica
I20260924 00:23:20.134635 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 13204
I20260924 00:23:20.134670 13507 raft_consensus.cc:2229] T c7f3d95e9180443abecaac1c92f43ab8 P b827fb68e9a44201a364856308b8432b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:23:20.134797 13507 raft_consensus.cc:2258] T c7f3d95e9180443abecaac1c92f43ab8 P b827fb68e9a44201a364856308b8432b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:23:20.135294 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.134794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45067 (local address 127.0.0.1:40256)
0924 00:23:20.135214 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:20.135253 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397}
W20260924 00:23:20.135439 12993 catalog_manager.cc:4584] TS c68a46d873bb4eb39681360c4e010e18 (127.0.98.1:45067): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c7f3d95e9180443abecaac1c92f43ab8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45067: connect: Connection refused (error 111)
I20260924 00:23:20.136842 13507 ts_tablet_manager.cc:1905] T c7f3d95e9180443abecaac1c92f43ab8 P b827fb68e9a44201a364856308b8432b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:23:20.139128 13507 ts_tablet_manager.cc:1918] T c7f3d95e9180443abecaac1c92f43ab8 P b827fb68e9a44201a364856308b8432b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:23:20.139223 13507 log.cc:1196] T c7f3d95e9180443abecaac1c92f43ab8 P b827fb68e9a44201a364856308b8432b: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/c7f3d95e9180443abecaac1c92f43ab8
I20260924 00:23:20.139523 13507 ts_tablet_manager.cc:1939] T c7f3d95e9180443abecaac1c92f43ab8 P b827fb68e9a44201a364856308b8432b: Deleting consensus metadata
I20260924 00:23:20.140122 12993 catalog_manager.cc:4841] TS b827fb68e9a44201a364856308b8432b (127.0.98.3:37021): tablet c7f3d95e9180443abecaac1c92f43ab8 (table test-table [id=837fb37c92a64edeb98c6ad7ebde94a2]) successfully deleted
I20260924 00:23:20.141149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 13338
I20260924 00:23:20.146579 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 12976
I20260924 00:23:20.151788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36285
--webserver_interface=127.0.98.62
--webserver_port=44051
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:23:20.241350 13509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:20.241570 13509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:20.241596 13509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:20.242784 13509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:23:20.242828 13509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:20.242842 13509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:20.242856 13509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:23:20.242867 13509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:23:20.244400 13509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:20.244618 13509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:20.244797 13509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:20.254912 13516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:20.254912 13518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:20.255029 13509 server_base.cc:1034] running on GCE node
W20260924 00:23:20.254952 13515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:20.255559 13509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:20.255904 13509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:20.257095 13509 hybrid_clock.cc:648] HybridClock initialized: now 1790209400257073 us; error 35 us; skew 500 ppm
Sep 24 00:23:20 dist-test-slave-777x krb5kdc[12963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:20.259863 13509 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:23:20.260457 13509 webserver.cc:460] Webserver started at http://127.0.98.62:44051/ using document root <none> and password file <none>
I20260924 00:23:20.260730 13509 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:20.260821 13509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:20.262575 13509 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:23:20.263459 13525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:20.263725 13509 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:23:20.263855 13509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "5433edd2f6ce47dcb09947233e09fd0c"
format_stamp: "Formatted at 2026-09-24 00:23:19 on dist-test-slave-777x"
server_key: "85bc76a6fe59a976bdb1339969d3ac83"
server_key_iv: "aac536a41a9230797a9399f586d9558e"
server_key_version: "encryptionkey@0"
I20260924 00:23:20.264364 13509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:20.273522 13509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:20.276486 13509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:20.276684 13509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:20.281144 13509 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36285
I20260924 00:23:20.281201 13577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36285 every 8 connection(s)
I20260924 00:23:20.281658 13509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:23:20.283435 13578 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:23:20.284222 13578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c: Bootstrap starting.
I20260924 00:23:20.287503 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 13509
I20260924 00:23:20.287855 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45067
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36285
--webserver_port=37471
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:23:20.293406 13578 log.cc:824] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:20.295881 13578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:23:20.296319 13578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c: Bootstrap complete.
I20260924 00:23:20.299019 13578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5433edd2f6ce47dcb09947233e09fd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } }
I20260924 00:23:20.299340 13578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5433edd2f6ce47dcb09947233e09fd0c, State: Initialized, Role: FOLLOWER
I20260924 00:23:20.299543 13578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5433edd2f6ce47dcb09947233e09fd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } }
I20260924 00:23:20.299625 13578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:23:20.299698 13578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:23:20.299752 13578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:23:20.301097 13578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5433edd2f6ce47dcb09947233e09fd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } }
I20260924 00:23:20.301234 13578 leader_election.cc:302] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5433edd2f6ce47dcb09947233e09fd0c; no voters:
I20260924 00:23:20.301481 13578 leader_election.cc:288] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 00:23:20.301524 13582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:23:20.301815 13582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 2 LEADER]: Becoming Leader. State: Replica: 5433edd2f6ce47dcb09947233e09fd0c, State: Running, Role: LEADER
I20260924 00:23:20.301926 13578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:23:20.302039 13582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5433edd2f6ce47dcb09947233e09fd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } }
I20260924 00:23:20.302501 13582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5433edd2f6ce47dcb09947233e09fd0c. Latest consensus state: current_term: 2 leader_uuid: "5433edd2f6ce47dcb09947233e09fd0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5433edd2f6ce47dcb09947233e09fd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } } }
I20260924 00:23:20.302498 13583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5433edd2f6ce47dcb09947233e09fd0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5433edd2f6ce47dcb09947233e09fd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } } }
I20260924 00:23:20.302799 13583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [sys.catalog]: This master's current role is: LEADER
I20260924 00:23:20.303213 13582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [sys.catalog]: This master's current role is: LEADER
I20260924 00:23:20.303238 13591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:23:20.304112 13591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:23:20.304286 13591 catalog_manager.cc:1259] Loaded cluster ID: a2e4f7ae143946d194025989511b750c
I20260924 00:23:20.304579 13591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:23:20.304893 13591 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 00:23:20.310186 13591 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 00:23:20.312918 13591 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:23:20.313139 13591 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c: Loaded TSK: 0
I20260924 00:23:20.313354 13591 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 00:23:20.377825 13580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:20.378053 13580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:20.378083 13580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:20.379343 13580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:20.379386 13580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:20.379415 13580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:23:20.380956 13580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45067
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37471
--tserver_master_addrs=127.0.98.62:36285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:20.381250 13580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:20.381500 13580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:20.386750 13605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:20.386773 13608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:20.386777 13606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:20.387602 13580 server_base.cc:1034] running on GCE node
I20260924 00:23:20.387840 13580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:20.388157 13580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:20.389328 13580 hybrid_clock.cc:648] HybridClock initialized: now 1790209400389309 us; error 34 us; skew 500 ppm
Sep 24 00:23:20 dist-test-slave-777x krb5kdc[12963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:20.392230 13580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:23:20.392786 13580 webserver.cc:460] Webserver started at http://127.0.98.1:37471/ using document root <none> and password file <none>
I20260924 00:23:20.393074 13580 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:20.393169 13580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:20.394796 13580 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:23:20.395673 13615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:20.395915 13580 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:23:20.396020 13580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "c68a46d873bb4eb39681360c4e010e18"
format_stamp: "Formatted at 2026-09-24 00:23:19 on dist-test-slave-777x"
server_key: "2efeee16bceb925c8602d986365d9fb1"
server_key_iv: "7ab21b57d3f0f3ff5932d160fec85dd3"
server_key_version: "encryptionkey@0"
I20260924 00:23:20.396512 13580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:20.426551 13580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:20.429955 13580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:20.430140 13580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:20.430450 13580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:23:20.431007 13622 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 00:23:20.432780 13580 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 00:23:20.432849 13580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 00:23:20.432891 13580 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 00:23:20.433856 13580 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 00:23:20.433907 13580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 00:23:20.434021 13622 tablet_bootstrap.cc:492] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18: Bootstrap starting.
I20260924 00:23:20.441789 13580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45067
I20260924 00:23:20.442183 13729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45067 every 8 connection(s)
I20260924 00:23:20.442279 13580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:23:20.444118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 13580
I20260924 00:23:20.444526 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39327
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36285
--webserver_port=41759
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:23:20.445451 13622 log.cc:824] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:20.450767 13622 tablet_bootstrap.cc:492] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:23:20.451164 13622 tablet_bootstrap.cc:492] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18: Bootstrap complete.
I20260924 00:23:20.452474 13622 ts_tablet_manager.cc:1397] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.002s
Sep 24 00:23:20 dist-test-slave-777x krb5kdc[12963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:23:20.454496 13622 raft_consensus.cc:348] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:20.454918 13622 raft_consensus.cc:729] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68a46d873bb4eb39681360c4e010e18, State: Initialized, Role: FOLLOWER
I20260924 00:23:20.455120 13622 consensus_queue.cc:260] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:20.455472 13622 ts_tablet_manager.cc:1428] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260924 00:23:20.459707 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.443206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.1:54917)
0924 00:23:20.443434 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:20.443487 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:20.444134 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:20.444816 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:20.444820 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:20.445196 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:20.445498 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.445505 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.447432 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.447434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.448839 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.448843 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.449907 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.449908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.450073 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.452376 (+ 2303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:20.452398 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:20.454589 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:20.457443 (+ 2854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:20.457451 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:20.457453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:20.457868 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:20.458210 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:20.458211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:20.458212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:20.458439 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:20.458987 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:20.458993 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:20.459169 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 00:23:20.459282 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 00:23:20.459476 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":120,"threads_started":1}
I20260924 00:23:20.459764 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.443329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54917 (local address 127.0.98.62:36285)
0924 00:23:20.443589 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:20.443604 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:20.443655 (+ 51us) server_negotiation.cc:407] Connection header received
0924 00:23:20.444482 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:20.444488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:20.444547 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:20.444674 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:20.445641 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.447275 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.448953 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.449738 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.449897 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.454729 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:20.454734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:20.454734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:20.454761 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:20.457316 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:20.457983 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:20.457986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:20.457987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:20.458108 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:20.458550 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:20.458552 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:20.458552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:20.458897 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:20.459002 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:20.459383 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 00:23:20.459548 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":106,"threads_started":1}
I20260924 00:23:20.461009 13730 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36285
I20260924 00:23:20.461117 13730 heartbeater.cc:464] Registering TS with master...
I20260924 00:23:20.461305 13730 heartbeater.cc:511] Master 127.0.98.62:36285 requested a full tablet report, sending...
I20260924 00:23:20.462177 13542 ts_manager.cc:194] Registered new tserver with Master: c68a46d873bb4eb39681360c4e010e18 (127.0.98.1:45067)
I20260924 00:23:20.463414 13542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54917
I20260924 00:23:20.464628 13730 heartbeater.cc:503] Master 127.0.98.62:36285 was elected leader, sending a full tablet report...
I20260924 00:23:20.469693 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.463920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45067 (local address 127.0.0.1:40266)
0924 00:23:20.464410 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:20.464462 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:20.464536 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:20.465253 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:20.465257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:20.465259 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:23:20.465333 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.465335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.466877 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.466879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.468321 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.468326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.469355 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.469356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.469464 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.469480 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 00:23:20.469536 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 00:23:20.469597 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":129,"threads_started":1}
I20260924 00:23:20.470049 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.464812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40266 (local address 127.0.98.1:45067)
0924 00:23:20.465024 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:20.465029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:20.465042 (+ 13us) server_negotiation.cc:407] Connection header received
0924 00:23:20.465113 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:20.465115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:20.465132 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:20.465307 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:23:20.465461 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.466741 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.468490 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.469234 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.469826 (+ 592us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.469901 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:20.469930 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 00:23:20.469976 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":105,"threads_started":1}
I20260924 00:23:20.470479 13664 tablet_service.cc:1514] Processing DeleteTablet for tablet c7f3d95e9180443abecaac1c92f43ab8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:23:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40266
I20260924 00:23:20.471045 13739 tablet_replica.cc:330] stopping tablet replica
I20260924 00:23:20.471163 13739 raft_consensus.cc:2229] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:23:20.471237 13739 raft_consensus.cc:2258] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:23:20.471568 13739 ts_tablet_manager.cc:1905] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:23:20.473871 13739 ts_tablet_manager.cc:1918] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:23:20.474042 13739 log.cc:1196] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/wal/wals/c7f3d95e9180443abecaac1c92f43ab8
I20260924 00:23:20.474327 13739 ts_tablet_manager.cc:1939] T c7f3d95e9180443abecaac1c92f43ab8 P c68a46d873bb4eb39681360c4e010e18: Deleting consensus metadata
I20260924 00:23:20.474879 13527 catalog_manager.cc:4841] TS c68a46d873bb4eb39681360c4e010e18 (127.0.98.1:45067): tablet c7f3d95e9180443abecaac1c92f43ab8 (table test-table [id=837fb37c92a64edeb98c6ad7ebde94a2]) successfully deleted
W20260924 00:23:20.538117 13735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:20.538388 13735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:20.538425 13735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:20.539677 13735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:20.539750 13735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:20.539795 13735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:23:20.541343 13735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41759
--tserver_master_addrs=127.0.98.62:36285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:20.541635 13735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:20.541896 13735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:20.547133 13745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:20.547133 13747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:20.547313 13744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:20.547607 13735 server_base.cc:1034] running on GCE node
I20260924 00:23:20.547856 13735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:20.548131 13735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:20.549364 13735 hybrid_clock.cc:648] HybridClock initialized: now 1790209400549348 us; error 27 us; skew 500 ppm
Sep 24 00:23:20 dist-test-slave-777x krb5kdc[12963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:20.552335 13735 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:23:20.552863 13735 webserver.cc:460] Webserver started at http://127.0.98.2:41759/ using document root <none> and password file <none>
I20260924 00:23:20.553119 13735 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:20.553210 13735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:20.554908 13735 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 00:23:20.555691 13754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:20.555876 13735 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:23:20.555976 13735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "c702a480526c43919e56ce508cab9167"
format_stamp: "Formatted at 2026-09-24 00:23:19 on dist-test-slave-777x"
server_key: "7105cb485a44ae200b7f5c19ca750aa7"
server_key_iv: "585f9f41b930a0c52bbbe87c4ee0075f"
server_key_version: "encryptionkey@0"
I20260924 00:23:20.556442 13735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:20.585209 13735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:20.587760 13735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:20.587944 13735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:20.588243 13735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:23:20.588596 13735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:23:20.588651 13735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:20.588696 13735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:23:20.588722 13735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:20.594985 13735 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39327
I20260924 00:23:20.595038 13867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39327 every 8 connection(s)
I20260924 00:23:20.595382 13735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:23:20.600797 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 13735
I20260924 00:23:20.601133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37021
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36285
--webserver_port=39053
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 00:23:20 dist-test-slave-777x krb5kdc[12963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:23:20.607245 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.596193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52755 (local address 127.0.98.62:36285)
0924 00:23:20.596269 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:20.596272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:20.596450 (+ 178us) server_negotiation.cc:407] Connection header received
0924 00:23:20.597064 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:20.597065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:20.597083 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:20.597142 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:20.598277 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.598843 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.599993 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.600299 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.600393 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.603457 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:20.603459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:20.603460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:20.603470 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:20.605329 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:20.605970 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:20.605971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:20.605972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:20.606094 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:20.606448 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:20.606449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:20.606449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:20.606571 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:20.606628 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:20.607086 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 00:23:20.607170 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 00:23:20.607601 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.596069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:52755)
0924 00:23:20.596284 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:20.596343 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:20.596944 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:20.597269 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:20.597273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:20.597904 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:20.598154 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.598160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.598963 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.598964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.599868 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.599871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.600406 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.600407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.600556 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.601279 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:20.601336 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:20.603344 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:20.605443 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:20.605454 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:20.605458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:20.605881 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:20.606176 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:20.606177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:20.606178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:20.606368 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:20.606644 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:20.606651 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:20.606868 (+ 217us) client_negotiation.cc:769] Sending connection context
0924 00:23:20.607019 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 00:23:20.607259 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":125,"threads_started":1}
I20260924 00:23:20.608261 13868 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36285
I20260924 00:23:20.608390 13868 heartbeater.cc:464] Registering TS with master...
I20260924 00:23:20.608640 13868 heartbeater.cc:511] Master 127.0.98.62:36285 requested a full tablet report, sending...
I20260924 00:23:20.609071 13542 ts_manager.cc:194] Registered new tserver with Master: c702a480526c43919e56ce508cab9167 (127.0.98.2:39327)
I20260924 00:23:20.609475 13542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52755
W20260924 00:23:20.695855 13871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:20.696086 13871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:20.696111 13871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:20.697319 13871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:20.697366 13871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:20.697393 13871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:23:20.698863 13871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39053
--tserver_master_addrs=127.0.98.62:36285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:20.699108 13871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:20.699359 13871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:20.704839 13879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:20.704809 13877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:20.704818 13876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:20.705075 13871 server_base.cc:1034] running on GCE node
I20260924 00:23:20.705331 13871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:20.705581 13871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:20.706759 13871 hybrid_clock.cc:648] HybridClock initialized: now 1790209400706734 us; error 56 us; skew 500 ppm
Sep 24 00:23:20 dist-test-slave-777x krb5kdc[12963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:20.709354 13871 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:23:20.709848 13871 webserver.cc:460] Webserver started at http://127.0.98.3:39053/ using document root <none> and password file <none>
I20260924 00:23:20.710069 13871 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:20.710156 13871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:20.711534 13871 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:23:20.712287 13886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:20.712497 13871 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:23:20.712610 13871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "b827fb68e9a44201a364856308b8432b"
format_stamp: "Formatted at 2026-09-24 00:23:19 on dist-test-slave-777x"
server_key: "8f9f1528f2063b5d3210f4374d0f189c"
server_key_iv: "9cc0d81a33b052d3fe0d6946317a60f9"
server_key_version: "encryptionkey@0"
I20260924 00:23:20.713056 13871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:20.731760 13871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:20.734210 13871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:20.734388 13871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:20.734642 13871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:23:20.734954 13871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:23:20.735010 13871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:20.735054 13871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:23:20.735085 13871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:20.741405 13871 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37021
I20260924 00:23:20.741446 13999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37021 every 8 connection(s)
I20260924 00:23:20.741830 13871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:23:20.747993 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 13871
Sep 24 00:23:20 dist-test-slave-777x krb5kdc[12963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:23:20.752347 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.742784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52897 (local address 127.0.98.62:36285)
0924 00:23:20.742844 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:20.742846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:20.743023 (+ 177us) server_negotiation.cc:407] Connection header received
0924 00:23:20.743542 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:20.743544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:20.743562 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:20.743637 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:20.744671 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.745224 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.746214 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.746419 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.746512 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.748991 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:20.748992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:20.748993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:20.749002 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:20.750751 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:20.751176 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:20.751177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:20.751178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:20.751248 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:20.751619 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:20.751620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:20.751621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:20.751738 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:20.751852 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:20.752146 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 00:23:20.752256 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:23:20.752660 14002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.742741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:52897)
0924 00:23:20.742891 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:20.742934 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:20.743399 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:20.743707 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:20.743710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:20.744366 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:20.744576 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.744581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.745309 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.745310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.746108 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.746110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.746507 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.746508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.746613 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.747141 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:20.747150 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:20.748885 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:20.750840 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:20.750846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:20.750848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:20.751096 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:20.751342 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:20.751343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:20.751343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:20.751504 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:20.751846 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:20.751852 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:20.751996 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 00:23:20.752110 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 00:23:20.752295 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260924 00:23:20.753397 14000 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36285
I20260924 00:23:20.753536 14000 heartbeater.cc:464] Registering TS with master...
I20260924 00:23:20.753742 14000 heartbeater.cc:511] Master 127.0.98.62:36285 requested a full tablet report, sending...
I20260924 00:23:20.754199 13542 ts_manager.cc:194] Registered new tserver with Master: b827fb68e9a44201a364856308b8432b (127.0.98.3:37021)
I20260924 00:23:20.754698 13542 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52897
I20260924 00:23:20.756420 14004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.748416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59480 (local address 127.0.98.62:36285)
0924 00:23:20.748545 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:20.748547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:20.748613 (+ 66us) server_negotiation.cc:407] Connection header received
0924 00:23:20.748684 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:20.748685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:20.748698 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:20.748762 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:20.749677 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.750335 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.751482 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.751748 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.751838 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.753338 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:20.753339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:20.753339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:20.753346 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:20.755098 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:20.755530 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:20.755531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:20.755531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:20.755592 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:20.755824 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:20.755825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:20.755825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:20.755938 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:20.756139 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:20.756258 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 00:23:20.756350 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":55,"threads_started":1}
I20260924 00:23:20.757318 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:23:20.760757 14004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.757705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59490 (local address 127.0.98.62:36285)
0924 00:23:20.757775 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:20.757777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:20.757881 (+ 104us) server_negotiation.cc:407] Connection header received
0924 00:23:20.757986 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:20.757987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:20.758029 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:20.758105 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:23:20.758232 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.758745 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.759695 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.759918 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.760015 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.760219 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:23:20.760394 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:23:20.760482 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:20.760655 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 00:23:20.760682 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 00:23:20.768253 13542 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:23:20.770613 13542 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:23:20.773337 13664 tablet_service.cc:1467] Processing CreateTablet for tablet 930f5d32b44c4473a925345a46e68d0b (DEFAULT_TABLE table=test-table [id=b5216bee39614d769ba33352e5f38e88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:23:20.773588 13664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930f5d32b44c4473a925345a46e68d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:23:20.775111 13622 tablet_bootstrap.cc:492] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18: Bootstrap starting.
I20260924 00:23:20.775491 13622 tablet_bootstrap.cc:654] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18: Neither blocks nor log segments found. Creating new log.
I20260924 00:23:20.776129 13622 tablet_bootstrap.cc:492] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18: No bootstrap required, opened a new log
I20260924 00:23:20.776225 13622 ts_tablet_manager.cc:1397] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 00:23:20.776356 13622 raft_consensus.cc:348] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:20.776449 13622 raft_consensus.cc:374] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:23:20.776500 13622 raft_consensus.cc:729] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68a46d873bb4eb39681360c4e010e18, State: Initialized, Role: FOLLOWER
I20260924 00:23:20.776569 13622 consensus_queue.cc:260] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:20.776688 13622 ts_tablet_manager.cc:1428] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:20.777705 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.772946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37021 (local address 127.0.0.1:48884)
0924 00:23:20.773220 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:20.773233 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:20.773283 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:20.773559 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:20.773560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:20.773562 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:23:20.773636 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.773638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.774802 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.774804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.775983 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.775987 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.777446 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.777447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.777522 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.777527 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:23:20.777585 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 00:23:20.777625 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251}
I20260924 00:23:20.778028 14007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.773040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48884 (local address 127.0.98.3:37021)
0924 00:23:20.773253 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:20.773259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:20.773280 (+ 21us) server_negotiation.cc:407] Connection header received
0924 00:23:20.773403 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:20.773406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:20.773428 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:20.773563 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:23:20.773697 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.774693 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.776081 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.777331 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.777409 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.777473 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:20.777624 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 00:23:20.777669 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":99,"threads_started":1}
I20260924 00:23:20.778841 13934 tablet_service.cc:1467] Processing CreateTablet for tablet 930f5d32b44c4473a925345a46e68d0b (DEFAULT_TABLE table=test-table [id=b5216bee39614d769ba33352e5f38e88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:23:20.779223 13934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930f5d32b44c4473a925345a46e68d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:23:20.779651 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.772993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39327 (local address 127.0.0.1:50486)
0924 00:23:20.773408 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:20.773417 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:20.773457 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:20.775617 (+ 2160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:20.775618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:20.775621 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:23:20.775676 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.775677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.777177 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.777178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.778469 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:20.778472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.779455 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.779456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:20.779505 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.779509 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:23:20.779562 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 00:23:20.779591 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":91,"threads_started":1}
I20260924 00:23:20.779716 14008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:20.775187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50486 (local address 127.0.98.2:39327)
0924 00:23:20.775369 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:20.775374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:20.775388 (+ 14us) server_negotiation.cc:407] Connection header received
0924 00:23:20.775430 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:20.775433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:20.775449 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:20.775531 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:23:20.775791 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.777051 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.778576 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:20.779369 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:20.779455 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:20.779514 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:20.779577 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 00:23:20.779629 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":96,"threads_started":1}
I20260924 00:23:20.780478 13802 tablet_service.cc:1467] Processing CreateTablet for tablet 930f5d32b44c4473a925345a46e68d0b (DEFAULT_TABLE table=test-table [id=b5216bee39614d769ba33352e5f38e88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:23:20.780781 13802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930f5d32b44c4473a925345a46e68d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:23:20.782562 14010 tablet_bootstrap.cc:492] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: Bootstrap starting.
I20260924 00:23:20.783242 14010 tablet_bootstrap.cc:654] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: Neither blocks nor log segments found. Creating new log.
I20260924 00:23:20.783854 14010 log.cc:824] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:20.784089 14011 tablet_bootstrap.cc:492] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167: Bootstrap starting.
I20260924 00:23:20.784605 14010 tablet_bootstrap.cc:492] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: No bootstrap required, opened a new log
I20260924 00:23:20.784687 14010 ts_tablet_manager.cc:1397] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:23:20.784756 14011 tablet_bootstrap.cc:654] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167: Neither blocks nor log segments found. Creating new log.
I20260924 00:23:20.785418 14011 log.cc:824] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:20.786158 14011 tablet_bootstrap.cc:492] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167: No bootstrap required, opened a new log
I20260924 00:23:20.786211 14011 ts_tablet_manager.cc:1397] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 00:23:20.786405 14010 raft_consensus.cc:348] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:20.786566 14010 raft_consensus.cc:374] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:23:20.786612 14010 raft_consensus.cc:729] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b827fb68e9a44201a364856308b8432b, State: Initialized, Role: FOLLOWER
I20260924 00:23:20.786746 14010 consensus_queue.cc:260] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:20.787039 14010 ts_tablet_manager.cc:1428] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 00:23:20.787089 14000 heartbeater.cc:503] Master 127.0.98.62:36285 was elected leader, sending a full tablet report...
I20260924 00:23:20.788118 14011 raft_consensus.cc:348] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:20.788352 14011 raft_consensus.cc:374] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:23:20.788379 14011 raft_consensus.cc:729] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c702a480526c43919e56ce508cab9167, State: Initialized, Role: FOLLOWER
I20260924 00:23:20.788483 14011 consensus_queue.cc:260] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:20.788719 14011 ts_tablet_manager.cc:1428] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260924 00:23:20.788765 13868 heartbeater.cc:503] Master 127.0.98.62:36285 was elected leader, sending a full tablet report...
W20260924 00:23:20.846081 13869 tablet.cc:2369] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:23:20.944422 13731 tablet.cc:2369] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:23:20.992507 14001 tablet.cc:2369] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:23:21.131968 13736 raft_consensus.cc:482] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:21.132128 13736 raft_consensus.cc:504] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:21.132515 13736 leader_election.cc:288] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b827fb68e9a44201a364856308b8432b (127.0.98.3:37021), c702a480526c43919e56ce508cab9167 (127.0.98.2:39327)
I20260924 00:23:21.135746 14008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:21.132826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57169 (local address 127.0.98.2:39327)
0924 00:23:21.132889 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:21.132891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:21.132941 (+ 50us) server_negotiation.cc:407] Connection header received
0924 00:23:21.133017 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:21.133018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:21.133030 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:21.133088 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:23:21.133300 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:21.133876 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:21.134967 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:21.135355 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:21.135410 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:21.135420 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:21.135622 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 00:23:21.135672 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 00:23:21.135749 14016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:21.132712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39327 (local address 127.0.98.1:57169)
0924 00:23:21.132869 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:21.132884 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:21.132957 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:21.133105 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:21.133106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:21.133110 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:23:21.133204 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:21.133206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:21.134005 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:21.134006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:21.134860 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:21.134862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:21.135444 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:21.135445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:21.135538 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:21.135543 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:23:21.135599 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 00:23:21.135637 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":32,"thread_start_us":82,"threads_started":1}
I20260924 00:23:21.135874 14007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:21.132816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45029 (local address 127.0.98.3:37021)
0924 00:23:21.132899 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:21.132902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:21.132940 (+ 38us) server_negotiation.cc:407] Connection header received
0924 00:23:21.133088 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:21.133089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:21.133102 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:21.133203 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:23:21.133416 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:21.133865 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:21.135025 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:21.135438 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:21.135511 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:21.135520 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:21.135748 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 00:23:21.135798 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 00:23:21.135890 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:21.132718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37021 (local address 127.0.98.1:45029)
0924 00:23:21.132877 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:21.132889 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:21.132958 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:21.133268 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:21.133269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:21.133272 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:23:21.133336 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:21.133338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:21.134023 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:21.134025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:21.134873 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:21.134875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:21.135526 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:21.135527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:21.135611 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:21.135616 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:23:21.135681 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 00:23:21.135723 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":36,"thread_start_us":80,"threads_started":1}
I20260924 00:23:21.136096 13822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930f5d32b44c4473a925345a46e68d0b" candidate_uuid: "c68a46d873bb4eb39681360c4e010e18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c702a480526c43919e56ce508cab9167" is_pre_election: true
I20260924 00:23:21.136281 13822 raft_consensus.cc:2454] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c68a46d873bb4eb39681360c4e010e18 in term 0.
I20260924 00:23:21.136240 13954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930f5d32b44c4473a925345a46e68d0b" candidate_uuid: "c68a46d873bb4eb39681360c4e010e18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b827fb68e9a44201a364856308b8432b" is_pre_election: true
I20260924 00:23:21.136406 13954 raft_consensus.cc:2454] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c68a46d873bb4eb39681360c4e010e18 in term 0.
I20260924 00:23:21.136523 13616 leader_election.cc:302] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c68a46d873bb4eb39681360c4e010e18, c702a480526c43919e56ce508cab9167; no voters:
I20260924 00:23:21.136664 13736 raft_consensus.cc:2781] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:23:21.136729 13736 raft_consensus.cc:482] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:21.136765 13736 raft_consensus.cc:3037] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:23:21.137498 13736 raft_consensus.cc:504] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:21.137651 13736 leader_election.cc:288] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [CANDIDATE]: Term 1 election: Requested vote from peers b827fb68e9a44201a364856308b8432b (127.0.98.3:37021), c702a480526c43919e56ce508cab9167 (127.0.98.2:39327)
I20260924 00:23:21.137849 13954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930f5d32b44c4473a925345a46e68d0b" candidate_uuid: "c68a46d873bb4eb39681360c4e010e18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b827fb68e9a44201a364856308b8432b"
I20260924 00:23:21.137893 13822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930f5d32b44c4473a925345a46e68d0b" candidate_uuid: "c68a46d873bb4eb39681360c4e010e18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c702a480526c43919e56ce508cab9167"
I20260924 00:23:21.137959 13954 raft_consensus.cc:3037] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:23:21.137964 13822 raft_consensus.cc:3037] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:23:21.138706 13954 raft_consensus.cc:2454] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c68a46d873bb4eb39681360c4e010e18 in term 1.
I20260924 00:23:21.138777 13822 raft_consensus.cc:2454] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c68a46d873bb4eb39681360c4e010e18 in term 1.
I20260924 00:23:21.138950 13617 leader_election.cc:302] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b827fb68e9a44201a364856308b8432b, c68a46d873bb4eb39681360c4e010e18; no voters:
I20260924 00:23:21.139048 13736 raft_consensus.cc:2781] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:23:21.139171 13736 raft_consensus.cc:686] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [term 1 LEADER]: Becoming Leader. State: Replica: c68a46d873bb4eb39681360c4e010e18, State: Running, Role: LEADER
I20260924 00:23:21.139240 13736 consensus_queue.cc:237] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:21.140054 13542 catalog_manager.cc:5495] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 reported cstate change: term changed from 0 to 1, leader changed from <none> to c68a46d873bb4eb39681360c4e010e18 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c68a46d873bb4eb39681360c4e010e18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } health_report { overall_health: HEALTHY } } }
I20260924 00:23:21.219460 14021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:21.216291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40280 (local address 127.0.98.1:45067)
0924 00:23:21.216519 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:21.216521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:21.216528 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:23:21.216568 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:21.216568 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:21.216588 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:21.216679 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:23:21.216792 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:21.217769 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:21.218551 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:21.218774 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:21.218853 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:21.219069 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:23:21.219177 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:23:21.219265 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:21.219309 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 00:23:21.219331 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":105,"threads_started":1}
I20260924 00:23:21.221381 13954 raft_consensus.cc:1261] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b [term 1 FOLLOWER]: Refusing update from remote peer c68a46d873bb4eb39681360c4e010e18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:23:21.221381 13822 raft_consensus.cc:1261] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [term 1 FOLLOWER]: Refusing update from remote peer c68a46d873bb4eb39681360c4e010e18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:23:21.221850 14018 consensus_queue.cc:1035] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:23:21.222015 13736 consensus_queue.cc:1035] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:23:21.224588 14022 mvcc.cc:205] Tried to move back new op lower bound from 7332697707399249920 to 7332697707066912768. Current Snapshot: MvccSnapshot[applied={T|T < 7332697707399249920}]
I20260924 00:23:21.224784 14023 mvcc.cc:205] Tried to move back new op lower bound from 7332697707399249920 to 7332697707066912768. Current Snapshot: MvccSnapshot[applied={T|T < 7332697707399249920}]
I20260924 00:23:21.225320 14024 mvcc.cc:205] Tried to move back new op lower bound from 7332697707399249920 to 7332697707066912768. Current Snapshot: MvccSnapshot[applied={T|T < 7332697707399249920}]
I20260924 00:23:21.227310 13542 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:59490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:23:21.227397 13542 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:59490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:23:21.228195 13542 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c: Sending DeleteTablet for 3 replicas of tablet 930f5d32b44c4473a925345a46e68d0b
I20260924 00:23:21.228557 13934 tablet_service.cc:1514] Processing DeleteTablet for tablet 930f5d32b44c4473a925345a46e68d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:23:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48884
I20260924 00:23:21.228576 13664 tablet_service.cc:1514] Processing DeleteTablet for tablet 930f5d32b44c4473a925345a46e68d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:23:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40266
I20260924 00:23:21.228592 13802 tablet_service.cc:1514] Processing DeleteTablet for tablet 930f5d32b44c4473a925345a46e68d0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:23:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50486
I20260924 00:23:21.228850 14031 tablet_replica.cc:330] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: stopping tablet replica
I20260924 00:23:21.228950 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 13580
I20260924 00:23:21.228976 14033 tablet_replica.cc:330] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18: stopping tablet replica
I20260924 00:23:21.229090 14033 raft_consensus.cc:2229] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:23:21.229868 14031 raft_consensus.cc:2229] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:23:21.230018 14031 raft_consensus.cc:2258] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:23:21.230232 14032 tablet_replica.cc:330] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167: stopping tablet replica
I20260924 00:23:21.230345 14032 raft_consensus.cc:2229] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:23:21.230479 14032 raft_consensus.cc:2258] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:23:21.230587 14031 ts_tablet_manager.cc:1905] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:23:21.232815 14031 ts_tablet_manager.cc:1918] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:23:21.232908 14031 log.cc:1196] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/930f5d32b44c4473a925345a46e68d0b
I20260924 00:23:21.233182 14031 ts_tablet_manager.cc:1939] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: Deleting consensus metadata
I20260924 00:23:21.233733 13527 catalog_manager.cc:4841] TS b827fb68e9a44201a364856308b8432b (127.0.98.3:37021): tablet 930f5d32b44c4473a925345a46e68d0b (table test-table [id=b5216bee39614d769ba33352e5f38e88]) successfully deleted
W20260924 00:23:21.235616 13527 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45067 (error 108)
I20260924 00:23:21.236133 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:21.235996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45067 (local address 127.0.0.1:40284)
0924 00:23:21.236047 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:21.236087 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45067: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 00:23:21.236328 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 13735
W20260924 00:23:21.236303 13527 catalog_manager.cc:4584] TS c68a46d873bb4eb39681360c4e010e18 (127.0.98.1:45067): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 930f5d32b44c4473a925345a46e68d0b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45067: connect: Connection refused (error 111)
I20260924 00:23:21.241204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 13871
I20260924 00:23:21.241417 14006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:21.241278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39327 (local address 127.0.0.1:50488)
0924 00:23:21.241344 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:21.241375 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 00:23:21.246151 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 13509
I20260924 00:23:21.252038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36285
--webserver_interface=127.0.98.62
--webserver_port=44051
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:23:21.337926 14035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:21.338207 14035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:21.338249 14035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:21.339479 14035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:23:21.339555 14035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:21.339576 14035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:21.339597 14035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:23:21.339628 14035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:23:21.341194 14035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:21.341490 14035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:21.341747 14035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:21.346794 14040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:21.346853 14043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:21.346871 14035 server_base.cc:1034] running on GCE node
W20260924 00:23:21.346794 14041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:21.347436 14035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:21.347746 14035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:21.348915 14035 hybrid_clock.cc:648] HybridClock initialized: now 1790209401348902 us; error 30 us; skew 500 ppm
Sep 24 00:23:21 dist-test-slave-777x krb5kdc[12963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:21.351799 14035 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:23:21.352442 14035 webserver.cc:460] Webserver started at http://127.0.98.62:44051/ using document root <none> and password file <none>
I20260924 00:23:21.352726 14035 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:21.352792 14035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:21.354272 14035 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:23:21.355329 14050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:21.355645 14035 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:23:21.355759 14035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "5433edd2f6ce47dcb09947233e09fd0c"
format_stamp: "Formatted at 2026-09-24 00:23:19 on dist-test-slave-777x"
server_key: "85bc76a6fe59a976bdb1339969d3ac83"
server_key_iv: "aac536a41a9230797a9399f586d9558e"
server_key_version: "encryptionkey@0"
I20260924 00:23:21.356257 14035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:21.373390 14035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:21.376152 14035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:21.376308 14035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:21.380649 14035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36285
I20260924 00:23:21.380681 14102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36285 every 8 connection(s)
I20260924 00:23:21.381098 14035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:23:21.382704 14103 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:23:21.383450 14103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c: Bootstrap starting.
I20260924 00:23:21.387439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 14035
I20260924 00:23:21.387751 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45067
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36285
--webserver_port=37471
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:23:21.392582 14103 log.cc:824] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:21.394974 14103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:23:21.395364 14103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c: Bootstrap complete.
I20260924 00:23:21.397773 14103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5433edd2f6ce47dcb09947233e09fd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } }
I20260924 00:23:21.398007 14103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5433edd2f6ce47dcb09947233e09fd0c, State: Initialized, Role: FOLLOWER
I20260924 00:23:21.398177 14103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5433edd2f6ce47dcb09947233e09fd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } }
I20260924 00:23:21.398269 14103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:23:21.398325 14103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:23:21.398391 14103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:23:21.399463 14103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5433edd2f6ce47dcb09947233e09fd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } }
I20260924 00:23:21.399595 14103 leader_election.cc:302] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5433edd2f6ce47dcb09947233e09fd0c; no voters:
I20260924 00:23:21.399802 14103 leader_election.cc:288] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 00:23:21.399843 14107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:23:21.399960 14107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [term 3 LEADER]: Becoming Leader. State: Replica: 5433edd2f6ce47dcb09947233e09fd0c, State: Running, Role: LEADER
I20260924 00:23:21.400058 14107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5433edd2f6ce47dcb09947233e09fd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } }
I20260924 00:23:21.400084 14103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:23:21.400398 14109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5433edd2f6ce47dcb09947233e09fd0c. Latest consensus state: current_term: 3 leader_uuid: "5433edd2f6ce47dcb09947233e09fd0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5433edd2f6ce47dcb09947233e09fd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } } }
I20260924 00:23:21.400486 14109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [sys.catalog]: This master's current role is: LEADER
I20260924 00:23:21.400530 14108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5433edd2f6ce47dcb09947233e09fd0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5433edd2f6ce47dcb09947233e09fd0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36285 } } }
I20260924 00:23:21.400722 14108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5433edd2f6ce47dcb09947233e09fd0c [sys.catalog]: This master's current role is: LEADER
I20260924 00:23:21.400969 14112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:23:21.401994 14112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:23:21.402151 14112 catalog_manager.cc:1259] Loaded cluster ID: a2e4f7ae143946d194025989511b750c
I20260924 00:23:21.402221 14112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 00:23:21.402686 14112 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f845a8099ad google::LogMessage::Fail() at ??:0
@ 0x7f845a80c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f845a8094de google::LogMessage::Flush() at ??:0
@ 0x7f845a80aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f846272e0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f84627378b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f845c6d70d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f845c6cc954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f845d5306db start_thread at ??:0
@ 0x7f845930761f clone at ??:0
W20260924 00:23:21.515882 14105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:21.516121 14105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:21.516148 14105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:21.517594 14105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:21.517654 14105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:21.517685 14105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:23:21.519373 14105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45067
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37471
--tserver_master_addrs=127.0.98.62:36285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:21.519661 14105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:21.519935 14105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:21.526485 14139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:21.526574 14140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:21.526500 14142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:21.527652 14105 server_base.cc:1034] running on GCE node
I20260924 00:23:21.527951 14105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:21.528347 14105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:21.529541 14105 hybrid_clock.cc:648] HybridClock initialized: now 1790209401529522 us; error 39 us; skew 500 ppm
Sep 24 00:23:21 dist-test-slave-777x krb5kdc[12963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:21.532450 14105 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:23:21.532992 14105 webserver.cc:460] Webserver started at http://127.0.98.1:37471/ using document root <none> and password file <none>
I20260924 00:23:21.533269 14105 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:21.533346 14105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:21.535108 14105 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:23:21.535935 14149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:21.536212 14105 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:23:21.536301 14105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "c68a46d873bb4eb39681360c4e010e18"
format_stamp: "Formatted at 2026-09-24 00:23:19 on dist-test-slave-777x"
server_key: "2efeee16bceb925c8602d986365d9fb1"
server_key_iv: "7ab21b57d3f0f3ff5932d160fec85dd3"
server_key_version: "encryptionkey@0"
I20260924 00:23:21.536700 14105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:21.567155 14105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:21.569736 14105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:21.569857 14105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:21.570173 14105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:23:21.570657 14156 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 00:23:21.572968 14105 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 00:23:21.573053 14105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260924 00:23:21.573170 14105 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 00:23:21.573838 14105 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 00:23:21.573896 14105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 00:23:21.573982 14156 tablet_bootstrap.cc:492] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18: Bootstrap starting.
I20260924 00:23:21.581632 14105 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45067
I20260924 00:23:21.582080 14105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:23:21.582607 14263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45067 every 8 connection(s)
I20260924 00:23:21.584847 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 14105
I20260924 00:23:21.585180 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39327
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36285
--webserver_port=41759
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:23:21.586534 14156 log.cc:824] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:21.589766 14156 tablet_bootstrap.cc:492] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:23:21.590178 14156 tablet_bootstrap.cc:492] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18: Bootstrap complete.
I20260924 00:23:21.591603 14156 ts_tablet_manager.cc:1397] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.004s
I20260924 00:23:21.593277 14156 raft_consensus.cc:348] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:21.593654 14156 raft_consensus.cc:729] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68a46d873bb4eb39681360c4e010e18, State: Initialized, Role: FOLLOWER
I20260924 00:23:21.593849 14156 consensus_queue.cc:260] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:21.594162 14156 ts_tablet_manager.cc:1428] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18: Time spent starting tablet: real 0.002s user 0.002s sys 0.004s
W20260924 00:23:21.661764 14266 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:23:21.582828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.1:48625)
0924 00:23:21.583037 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:21.583099 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:21.583668 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:21.661521 (+ 77853us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":131,"threads_started":1}
W20260924 00:23:21.662091 14264 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36285 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36285: Client connection negotiation failed: client connection to 127.0.98.62:36285: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 00:23:21.662493 14266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:21.662345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.1:52633)
0924 00:23:21.662403 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:21.662459 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 00:23:21.662860 14266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:21.662753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.1:59311)
0924 00:23:21.662807 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:21.662837 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 00:23:21.662997 14264 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 00:23:21.673756 14267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:21.673995 14267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:21.674024 14267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:21.675380 14267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:21.675431 14267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:21.675462 14267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:23:21.677310 14267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41759
--tserver_master_addrs=127.0.98.62:36285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:21.677599 14267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:21.677865 14267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:21.683449 14275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:21.683463 14274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:21.683455 14277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:21.683679 14267 server_base.cc:1034] running on GCE node
I20260924 00:23:21.683883 14267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:21.684146 14267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:21.685338 14267 hybrid_clock.cc:648] HybridClock initialized: now 1790209401685325 us; error 28 us; skew 500 ppm
Sep 24 00:23:21 dist-test-slave-777x krb5kdc[12963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:21.687909 14267 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:23:21.688508 14267 webserver.cc:460] Webserver started at http://127.0.98.2:41759/ using document root <none> and password file <none>
I20260924 00:23:21.688757 14267 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:21.688828 14267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:21.690313 14267 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:23:21.691082 14284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:21.691272 14267 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:23:21.691388 14267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "c702a480526c43919e56ce508cab9167"
format_stamp: "Formatted at 2026-09-24 00:23:19 on dist-test-slave-777x"
server_key: "7105cb485a44ae200b7f5c19ca750aa7"
server_key_iv: "585f9f41b930a0c52bbbe87c4ee0075f"
server_key_version: "encryptionkey@0"
I20260924 00:23:21.691756 14267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:21.705020 14267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:21.707348 14267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:21.707533 14267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:21.707790 14267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:23:21.708325 14291 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 00:23:21.709992 14267 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 00:23:21.710069 14267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 00:23:21.710121 14267 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 00:23:21.710778 14267 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 00:23:21.710834 14267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 00:23:21.710937 14291 tablet_bootstrap.cc:492] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167: Bootstrap starting.
I20260924 00:23:21.712785 14291 log.cc:824] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:21.715482 14291 tablet_bootstrap.cc:492] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:23:21.715817 14291 tablet_bootstrap.cc:492] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167: Bootstrap complete.
I20260924 00:23:21.716054 14291 ts_tablet_manager.cc:1397] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 00:23:21.718099 14267 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39327
I20260924 00:23:21.718185 14399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39327 every 8 connection(s)
I20260924 00:23:21.718487 14267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:23:21.718204 14291 raft_consensus.cc:348] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:21.718605 14291 raft_consensus.cc:729] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c702a480526c43919e56ce508cab9167, State: Initialized, Role: FOLLOWER
I20260924 00:23:21.718858 14291 consensus_queue.cc:260] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:21.719206 14291 ts_tablet_manager.cc:1428] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167: Time spent starting tablet: real 0.003s user 0.001s sys 0.005s
I20260924 00:23:21.719635 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:21.719178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:33251)
0924 00:23:21.719352 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:21.719457 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
W20260924 00:23:21.719868 14400 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36285 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36285: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
I20260924 00:23:21.720260 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:21.720077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:38533)
0924 00:23:21.720160 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:21.720238 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 00:23:21.720516 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:21.720457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:59657)
0924 00:23:21.720487 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:21.720505 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 00:23:21.720615 14400 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 00:23:21.721139 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 14267
I20260924 00:23:21.721341 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37021
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36285
--webserver_port=39053
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:23:21.806959 14404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:21.807191 14404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:21.807219 14404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:21.808419 14404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:21.808470 14404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:21.808501 14404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:23:21.809937 14404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37021
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39053
--tserver_master_addrs=127.0.98.62:36285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:21.810144 14404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:21.810334 14404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:21.816042 14412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:21.816109 14404 server_base.cc:1034] running on GCE node
W20260924 00:23:21.816229 14410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:21.816112 14409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:21.816561 14404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:21.816825 14404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:21.818071 14404 hybrid_clock.cc:648] HybridClock initialized: now 1790209401818050 us; error 58 us; skew 500 ppm
Sep 24 00:23:21 dist-test-slave-777x krb5kdc[12963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:21.821316 14404 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:23:21.821903 14404 webserver.cc:460] Webserver started at http://127.0.98.3:39053/ using document root <none> and password file <none>
I20260924 00:23:21.822189 14404 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:21.822261 14404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:21.823659 14404 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:23:21.824399 14419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:21.824602 14404 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:23:21.824709 14404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "b827fb68e9a44201a364856308b8432b"
format_stamp: "Formatted at 2026-09-24 00:23:19 on dist-test-slave-777x"
server_key: "8f9f1528f2063b5d3210f4374d0f189c"
server_key_iv: "9cc0d81a33b052d3fe0d6946317a60f9"
server_key_version: "encryptionkey@0"
I20260924 00:23:21.825089 14404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:21.840849 14404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:21.842793 14404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:21.842967 14404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:21.843214 14404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:23:21.843539 14404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:23:21.843593 14404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:21.843628 14404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:23:21.843657 14404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:21.849712 14404 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37021
I20260924 00:23:21.849761 14532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37021 every 8 connection(s)
I20260924 00:23:21.850111 14404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:23:21.851241 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:21.850812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:46301)
0924 00:23:21.851007 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:21.851097 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
W20260924 00:23:21.851763 14533 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36285 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36285: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
I20260924 00:23:21.852087 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:21.851983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:60099)
0924 00:23:21.852032 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:21.852063 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 00:23:21.852415 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:21.852348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:42539)
0924 00:23:21.852375 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:21.852399 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 00:23:21.852542 14533 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 00:23:21.856746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 14404
I20260924 00:23:22.663658 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:22.663386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.1:55967)
0924 00:23:22.663540 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:22.663587 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":64,"threads_started":1}
I20260924 00:23:22.721171 14538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:22.720959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:39009)
0924 00:23:22.721090 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:22.721125 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":56,"threads_started":1}
I20260924 00:23:22.853092 14539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:22.852868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:47279)
0924 00:23:22.852995 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:22.853042 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260924 00:23:22.949484 14540 raft_consensus.cc:482] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:22.949687 14540 raft_consensus.cc:504] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:22.950101 14540 leader_election.cc:288] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b827fb68e9a44201a364856308b8432b (127.0.98.3:37021), c68a46d873bb4eb39681360c4e010e18 (127.0.98.1:45067)
Sep 24 00:23:22 dist-test-slave-777x krb5kdc[12963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 24 00:23:22 dist-test-slave-777x krb5kdc[12963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:23:22.961380 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:22.950541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37413 (local address 127.0.98.3:37021)
0924 00:23:22.950768 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:22.950779 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:22.950808 (+ 29us) server_negotiation.cc:407] Connection header received
0924 00:23:22.951406 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:22.951409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:22.951459 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:22.951607 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:22.952036 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:22.952950 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:22.953997 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:22.954583 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:22.954699 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:22.957350 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:22.957355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:22.957355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:22.957369 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:22.959355 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:22.960123 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:22.960126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:22.960126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:22.960254 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:22.960513 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:22.960514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:22.960514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:22.960801 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:22.960885 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:22.961074 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 00:23:22.961264 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":60,"threads_started":1}
I20260924 00:23:22.961373 14538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:22.950368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37021 (local address 127.0.98.2:37413)
0924 00:23:22.950437 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:22.950467 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:22.950671 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:22.951609 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:22.951610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:22.951765 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:22.951893 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:22.951898 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:22.953069 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:22.953070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:22.953880 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:22.953882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:22.954688 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:22.954689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:22.954838 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:22.955417 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:22.955428 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:22.957227 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:22.959818 (+ 2591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:22.959819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:22.959819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:22.960009 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:22.960373 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:22.960374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:22.960375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:22.960437 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:22.960876 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:22.960877 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:22.960926 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 00:23:22.960995 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 00:23:22.961212 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":3}
I20260924 00:23:22.961448 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:22.950578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38399 (local address 127.0.98.1:45067)
0924 00:23:22.950730 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:22.950736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:22.950748 (+ 12us) server_negotiation.cc:407] Connection header received
0924 00:23:22.950850 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:22.950852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:22.950872 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:22.950938 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:22.952014 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:22.952977 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:22.953966 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:22.954584 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:22.954699 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:22.957659 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:22.957663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:22.957664 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:22.957675 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:22.959349 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:22.959805 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:22.959807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:22.959808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:22.959961 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:22.960299 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:22.960300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:22.960300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:22.960673 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:22.960738 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:22.961169 (+ 431us) server_negotiation.cc:299] Negotiation successful
0924 00:23:22.961331 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":57,"threads_started":1}
I20260924 00:23:22.961802 14541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:22.950472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45067 (local address 127.0.98.2:38399)
0924 00:23:22.950581 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:22.950592 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:22.950674 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:22.951008 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:22.951013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:22.951406 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:22.951893 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:22.951899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:22.953126 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:22.953127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:22.953884 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:22.953887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:22.954689 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:22.954690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:22.954838 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:22.955440 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:22.955444 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:22.957552 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:22.959444 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:22.959450 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:22.959452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:22.959703 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:22.960065 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:22.960066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:22.960066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:22.960231 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:22.960745 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:22.960751 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:22.960899 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 00:23:22.961026 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 00:23:22.961293 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":147,"thread_start_us":59,"threads_started":1}
W20260924 00:23:22.962270 14286 leader_election.cc:341] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b827fb68e9a44201a364856308b8432b (127.0.98.3:37021): Not found: Tablet not found: 930f5d32b44c4473a925345a46e68d0b
I20260924 00:23:22.962450 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930f5d32b44c4473a925345a46e68d0b" candidate_uuid: "c702a480526c43919e56ce508cab9167" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c68a46d873bb4eb39681360c4e010e18" is_pre_election: true
I20260924 00:23:22.962604 14218 raft_consensus.cc:2454] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c702a480526c43919e56ce508cab9167 in term 1.
I20260924 00:23:22.962881 14286 leader_election.cc:302] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c68a46d873bb4eb39681360c4e010e18, c702a480526c43919e56ce508cab9167; no voters: b827fb68e9a44201a364856308b8432b
I20260924 00:23:22.963016 14540 raft_consensus.cc:2781] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 00:23:22.963145 14540 raft_consensus.cc:482] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:22.963182 14540 raft_consensus.cc:3037] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:23:22.964488 14540 raft_consensus.cc:504] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:22.964630 14540 leader_election.cc:288] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [CANDIDATE]: Term 2 election: Requested vote from peers b827fb68e9a44201a364856308b8432b (127.0.98.3:37021), c68a46d873bb4eb39681360c4e010e18 (127.0.98.1:45067)
I20260924 00:23:22.964825 14218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930f5d32b44c4473a925345a46e68d0b" candidate_uuid: "c702a480526c43919e56ce508cab9167" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c68a46d873bb4eb39681360c4e010e18"
I20260924 00:23:22.964916 14218 raft_consensus.cc:3037] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [term 1 FOLLOWER]: Advancing to term 2
W20260924 00:23:22.965018 14286 leader_election.cc:341] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b827fb68e9a44201a364856308b8432b (127.0.98.3:37021): Not found: Tablet not found: 930f5d32b44c4473a925345a46e68d0b
I20260924 00:23:22.966050 14218 raft_consensus.cc:2454] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c702a480526c43919e56ce508cab9167 in term 2.
I20260924 00:23:22.966257 14286 leader_election.cc:302] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c68a46d873bb4eb39681360c4e010e18, c702a480526c43919e56ce508cab9167; no voters: b827fb68e9a44201a364856308b8432b
I20260924 00:23:22.966348 14540 raft_consensus.cc:2781] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:23:22.966470 14540 raft_consensus.cc:686] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [term 2 LEADER]: Becoming Leader. State: Replica: c702a480526c43919e56ce508cab9167, State: Running, Role: LEADER
I20260924 00:23:22.966571 14540 consensus_queue.cc:237] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:22.966894 14541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:22.966720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:45339)
0924 00:23:22.966812 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:22.966848 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260924 00:23:23.426338 14286 consensus_peers.cc:487] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 -> Peer b827fb68e9a44201a364856308b8432b (127.0.98.3:37021): Couldn't send request to peer b827fb68e9a44201a364856308b8432b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 930f5d32b44c4473a925345a46e68d0b. This is attempt 1: this message will repeat every 5th retry.
I20260924 00:23:23.578485 14218 raft_consensus.cc:1261] T 930f5d32b44c4473a925345a46e68d0b P c68a46d873bb4eb39681360c4e010e18 [term 2 FOLLOWER]: Refusing update from remote peer c702a480526c43919e56ce508cab9167: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260924 00:23:23.578950 14540 consensus_queue.cc:1035] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260924 00:23:23.580015 14548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:23.579800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.1:60113)
0924 00:23:23.579906 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:23.579964 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":45,"threads_started":1}
I20260924 00:23:23.581008 14548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:23.580901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.1:50687)
0924 00:23:23.580939 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:23.580973 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 00:23:23.581066 14552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:23.580795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:56115)
0924 00:23:23.580971 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:23.581008 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":62,"threads_started":1}
I20260924 00:23:23.853828 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:23.853599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:53981)
0924 00:23:23.853738 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:23.853781 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260924 00:23:23.951193 14554 ts_tablet_manager.cc:927] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: Initiating tablet copy from peer c702a480526c43919e56ce508cab9167 (127.0.98.2:39327)
I20260924 00:23:23.951470 14554 tablet_copy_client.cc:316] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.2:39327
Sep 24 00:23:23 dist-test-slave-777x krb5kdc[12963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 00:23:23.959878 14556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:23.951908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51021 (local address 127.0.98.2:39327)
0924 00:23:23.952110 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:23.952116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:23.952128 (+ 12us) server_negotiation.cc:407] Connection header received
0924 00:23:23.952157 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:23.952159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:23.952197 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:23.952266 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:23.952714 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:23.953463 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:23.954399 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:23.954713 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:23.954774 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:23.957022 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:23.957031 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:23.957031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:23.957038 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:23.958702 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:23.959061 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:23.959064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:23.959064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:23.959123 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:23.959355 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:23.959356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:23.959356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:23.959498 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:23.959564 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:23.959743 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 00:23:23.959810 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260924 00:23:23.959892 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:23.951809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39327 (local address 127.0.98.3:51021)
0924 00:23:23.951864 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:23.951893 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:23.951971 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:23.952272 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:23.952278 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:23.952563 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:23.952634 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:23.952636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:23.953558 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:23.953559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:23.954326 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:23.954329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:23.954779 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:23.954780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:23.954834 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:23.955361 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:23.955367 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:23.956942 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:23.958783 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:23.958789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:23.958791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:23.958985 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:23.959207 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:23.959208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:23.959209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:23.959280 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:23.959561 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:23.959569 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:23.959628 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 00:23:23.959703 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 00:23:23.959803 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 00:23:23.960462 14374 tablet_copy_service.cc:140] P c702a480526c43919e56ce508cab9167: Received BeginTabletCopySession request for tablet 930f5d32b44c4473a925345a46e68d0b from peer b827fb68e9a44201a364856308b8432b ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51021)
I20260924 00:23:23.960572 14374 tablet_copy_service.cc:161] P c702a480526c43919e56ce508cab9167: Beginning new tablet copy session on tablet 930f5d32b44c4473a925345a46e68d0b from peer b827fb68e9a44201a364856308b8432b at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51021: session id = b827fb68e9a44201a364856308b8432b-930f5d32b44c4473a925345a46e68d0b
I20260924 00:23:23.961232 14374 tablet_copy_source_session.cc:205] T 930f5d32b44c4473a925345a46e68d0b P c702a480526c43919e56ce508cab9167: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 00:23:23.962284 14554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930f5d32b44c4473a925345a46e68d0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:23:23.965111 14554 tablet_copy_client.cc:791] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: tablet copy: Starting download of 0 data blocks...
I20260924 00:23:23.965309 14554 tablet_copy_client.cc:655] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: tablet copy: Starting download of 1 WAL segments...
I20260924 00:23:23.966132 14554 tablet_copy_client.cc:531] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 00:23:23.966776 14554 tablet_bootstrap.cc:492] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: Bootstrap starting.
I20260924 00:23:23.968106 14554 log.cc:824] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:23.970563 14554 tablet_bootstrap.cc:492] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:23:23.970744 14554 tablet_bootstrap.cc:492] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: Bootstrap complete.
I20260924 00:23:23.970837 14554 ts_tablet_manager.cc:1397] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 00:23:23.971701 14554 raft_consensus.cc:348] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:23.971849 14554 raft_consensus.cc:729] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b827fb68e9a44201a364856308b8432b, State: Initialized, Role: FOLLOWER
I20260924 00:23:23.971977 14554 consensus_queue.cc:260] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b827fb68e9a44201a364856308b8432b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37021 } } peers { permanent_uuid: "c702a480526c43919e56ce508cab9167" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39327 } } peers { permanent_uuid: "c68a46d873bb4eb39681360c4e010e18" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45067 } }
I20260924 00:23:23.972273 14554 ts_tablet_manager.cc:1428] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20260924 00:23:23.972615 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:23.972467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:46381)
0924 00:23:23.972517 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:23.972576 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 00:23:23.972690 14374 tablet_copy_service.cc:342] P c702a480526c43919e56ce508cab9167: Request end of tablet copy session b827fb68e9a44201a364856308b8432b-930f5d32b44c4473a925345a46e68d0b received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51021
I20260924 00:23:23.972759 14374 tablet_copy_service.cc:433] P c702a480526c43919e56ce508cab9167: ending tablet copy session b827fb68e9a44201a364856308b8432b-930f5d32b44c4473a925345a46e68d0b on tablet 930f5d32b44c4473a925345a46e68d0b with peer b827fb68e9a44201a364856308b8432b
I20260924 00:23:23.972922 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:23.972850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:35845)
0924 00:23:23.972878 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:23.972901 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260924 00:23:24.516968 14561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:24.516770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:45661)
0924 00:23:24.516880 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:24.516924 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":42,"threads_started":1}
I20260924 00:23:24.516922 14487 raft_consensus.cc:1203] T 930f5d32b44c4473a925345a46e68d0b P b827fb68e9a44201a364856308b8432b [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260924 00:23:24.517556 14563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:24.517329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:41139)
0924 00:23:24.517445 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:24.517502 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260924 00:23:24.581612 14564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:24.581427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.1:50777)
0924 00:23:24.581529 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:24.581563 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":52,"threads_started":1}
I20260924 00:23:25.517759 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:25.517515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:44389)
0924 00:23:25.517672 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:25.517712 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260924 00:23:25.518229 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:25.518030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:57681)
0924 00:23:25.518150 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:25.518191 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260924 00:23:25.582528 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:25.582286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.1:45047)
0924 00:23:25.582438 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:25.582486 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":64,"threads_started":1}
I20260924 00:23:26.518637 14568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:26.518330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:33627)
0924 00:23:26.518514 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:26.518587 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
I20260924 00:23:26.518929 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:26.518645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:37483)
0924 00:23:26.518858 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:26.518892 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":78,"threads_started":1}
I20260924 00:23:26.583351 14570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:26.583120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.1:40071)
0924 00:23:26.583247 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:26.583292 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":92,"threads_started":1}
I20260924 00:23:27.519595 14573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:27.519320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:38831)
0924 00:23:27.519462 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:27.519520 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260924 00:23:27.519682 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:27.519474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:53019)
0924 00:23:27.519584 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:27.519619 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":66,"threads_started":1}
I20260924 00:23:27.520043 14573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:27.519945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:38729)
0924 00:23:27.519996 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:27.520024 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 00:23:27.583974 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:27.583774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.1:39027)
0924 00:23:27.583895 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:27.583929 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":50,"threads_started":1}
I20260924 00:23:28.520486 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:28.520228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:37313)
0924 00:23:28.520367 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:28.520420 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20260924 00:23:28.520872 14577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:28.520606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:51159)
0924 00:23:28.520791 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:28.520824 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":78,"threads_started":1}
I20260924 00:23:28.584679 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:28.584455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.1:58043)
0924 00:23:28.584589 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:28.584629 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260924 00:23:29.521252 14579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:29.520957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:57805)
0924 00:23:29.521121 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:29.521201 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":65,"threads_started":1}
I20260924 00:23:29.521593 14580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:29.521316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:59381)
0924 00:23:29.521488 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:29.521523 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":91,"threads_started":1}
I20260924 00:23:29.585427 14581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:29.585165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.1:34421)
0924 00:23:29.585341 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:29.585383 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":59,"threads_started":1}
I20260924 00:23:30.522243 14583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:30.522045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:43053)
0924 00:23:30.522168 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:30.522202 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20260924 00:23:30.522222 14582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:30.521879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:35971)
0924 00:23:30.522066 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:30.522115 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":102,"threads_started":1}
I20260924 00:23:30.586191 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:30.585973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.1:48211)
0924 00:23:30.586108 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:30.586148 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20260924 00:23:31.523051 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:31.522835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:46141)
0924 00:23:31.522965 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:31.523006 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260924 00:23:31.523065 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:31.522868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:59877)
0924 00:23:31.522981 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:31.523023 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":72,"threads_started":1}
I20260924 00:23:31.587103 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:31.586874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.1:51641)
0924 00:23:31.587006 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:31.587053 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260924 00:23:32.523927 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:32.523666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:41857)
0924 00:23:32.523820 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:32.523863 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260924 00:23:32.523945 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:32.523717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:41453)
0924 00:23:32.523857 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:32.523904 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260924 00:23:32.587854 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:32.587608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.1:39905)
0924 00:23:32.587757 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:32.587798 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":59,"threads_started":1}
I20260924 00:23:33.524685 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:33.524469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:35035)
0924 00:23:33.524607 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:33.524636 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260924 00:23:33.524690 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:33.524463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:34273)
0924 00:23:33.524598 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:33.524640 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260924 00:23:33.588642 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:33.588410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.1:41289)
0924 00:23:33.588553 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:33.588597 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":67,"threads_started":1}
I20260924 00:23:34.525372 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:34.525151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:43435)
0924 00:23:34.525278 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:34.525314 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260924 00:23:34.525403 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:34.525196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:38113)
0924 00:23:34.525313 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:34.525344 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
I20260924 00:23:34.589457 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:34.589174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.1:48797)
0924 00:23:34.589366 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:34.589406 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":63,"threads_started":1}
I20260924 00:23:35.526131 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:35.525880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:56511)
0924 00:23:35.526023 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:35.526074 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260924 00:23:35.526355 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:35.525875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:33171)
0924 00:23:35.526248 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:35.526308 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":79,"threads_started":1}
I20260924 00:23:35.590376 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:35.590106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.1:39133)
0924 00:23:35.590270 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:35.590315 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20260924 00:23:36.526980 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:36.526663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.3:38883)
0924 00:23:36.526877 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:36.526930 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":103,"threads_started":1}
I20260924 00:23:36.527021 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:36.526829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.2:57119)
0924 00:23:36.526941 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:36.526981 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":67,"threads_started":1}
I20260924 00:23:36.591334 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:36.591073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36285 (local address 127.0.98.1:55467)
0924 00:23:36.591238 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:36.591286 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":86,"threads_started":1}
I20260924 00:23:36.859090 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 14105
I20260924 00:23:36.864048 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 14267
I20260924 00:23:36.869395 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 14404
2026-09-24T00:23:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18825 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:23:36 dist-test-slave-777x krb5kdc[14605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:23:36 dist-test-slave-777x krb5kdc[14605](info): set up 2 sockets
Sep 24 00:23:36 dist-test-slave-777x krb5kdc[14605](info): commencing operation
krb5kdc: starting...
W20260924 00:23:37.924383 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:23:38 dist-test-slave-777x krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:23:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:23:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:23:38.055881 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37889
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:23:38.138929 14618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:38.139204 14618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:38.139240 14618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:38.140468 14618 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 00:23:38.140539 14618 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 00:23:38.140561 14618 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 00:23:38.140581 14618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:23:38.140612 14618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:38.140640 14618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:38.140661 14618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:23:38.140689 14618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:23:38.142221 14618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:38.142524 14618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:38.142773 14618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:38.148126 14624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:38.148126 14626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:38.148132 14623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:38.148584 14618 server_base.cc:1034] running on GCE node
I20260924 00:23:38.148855 14618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:38.149252 14618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:38.150445 14618 hybrid_clock.cc:648] HybridClock initialized: now 1790209418150428 us; error 42 us; skew 500 ppm
Sep 24 00:23:38 dist-test-slave-777x krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:38.153131 14618 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:23:38.153556 14618 webserver.cc:460] Webserver started at http://127.0.98.62:39923/ using document root <none> and password file <none>
I20260924 00:23:38.153743 14618 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:38.153784 14618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:38.153883 14618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:23:38.154953 14618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "f2aebaa49ce74c249f086f4d9e469185"
format_stamp: "Formatted at 2026-09-24 00:23:38 on dist-test-slave-777x"
server_key: "ecb3874a00faf02d94c6b15f69a48452"
server_key_iv: "de57ad7cc71d808701a4d47ac514f40e"
server_key_version: "encryptionkey@0"
I20260924 00:23:38.155268 14618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "f2aebaa49ce74c249f086f4d9e469185"
format_stamp: "Formatted at 2026-09-24 00:23:38 on dist-test-slave-777x"
server_key: "ecb3874a00faf02d94c6b15f69a48452"
server_key_iv: "de57ad7cc71d808701a4d47ac514f40e"
server_key_version: "encryptionkey@0"
I20260924 00:23:38.156697 14618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:23:38.157577 14633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:38.157802 14618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:23:38.157874 14618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "f2aebaa49ce74c249f086f4d9e469185"
format_stamp: "Formatted at 2026-09-24 00:23:38 on dist-test-slave-777x"
server_key: "ecb3874a00faf02d94c6b15f69a48452"
server_key_iv: "de57ad7cc71d808701a4d47ac514f40e"
server_key_version: "encryptionkey@0"
I20260924 00:23:38.157960 14618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:38.172636 14618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:38.175047 14618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:38.175205 14618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:38.179549 14618 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37889
I20260924 00:23:38.179586 14685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37889 every 8 connection(s)
I20260924 00:23:38.179960 14618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:23:38.180972 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 14618
I20260924 00:23:38.181115 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:23:38.181147 14686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:23:38.181331 392 external_mini_cluster.cc:1428] Setting key c699ad602ad0da07beec9b75438eae78
I20260924 00:23:38.184088 14686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185: Bootstrap starting.
I20260924 00:23:38.184739 14686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185: Neither blocks nor log segments found. Creating new log.
I20260924 00:23:38.185031 14686 log.cc:824] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:38.185740 14686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185: No bootstrap required, opened a new log
I20260924 00:23:38.187945 14686 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aebaa49ce74c249f086f4d9e469185" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } }
Sep 24 00:23:38 dist-test-slave-777x krb5kdc[14605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:23:38.188131 14686 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:23:38.188196 14686 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2aebaa49ce74c249f086f4d9e469185, State: Initialized, Role: FOLLOWER
I20260924 00:23:38.188351 14686 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aebaa49ce74c249f086f4d9e469185" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } }
I20260924 00:23:38.188453 14686 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:23:38.188499 14686 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:23:38.188549 14686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:23:38.189234 14686 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aebaa49ce74c249f086f4d9e469185" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } }
I20260924 00:23:38.189344 14686 leader_election.cc:302] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2aebaa49ce74c249f086f4d9e469185; no voters:
I20260924 00:23:38.189566 14686 leader_election.cc:288] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:23:38.189867 14686 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:23:38.189860 14691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:23:38.191351 14691 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 1 LEADER]: Becoming Leader. State: Replica: f2aebaa49ce74c249f086f4d9e469185, State: Running, Role: LEADER
I20260924 00:23:38.191505 14691 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aebaa49ce74c249f086f4d9e469185" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } }
I20260924 00:23:38.192072 14692 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2aebaa49ce74c249f086f4d9e469185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aebaa49ce74c249f086f4d9e469185" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } }
I20260924 00:23:38.192260 14692 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [sys.catalog]: This master's current role is: LEADER
I20260924 00:23:38.192467 14689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:38.181922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.0.98.62:37889)
0924 00:23:38.182137 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:38.182153 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:38.182180 (+ 27us) server_negotiation.cc:407] Connection header received
0924 00:23:38.182646 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:38.182652 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:38.182701 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:38.182853 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:38.183521 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.184530 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.185451 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.185920 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.186058 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:38.188740 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:38.188744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:38.188744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:38.188754 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:38.190663 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:38.191102 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:38.191105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:38.191105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:38.191208 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:38.191399 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:38.191400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:38.191401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:38.191731 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:38.191895 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:38.192028 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 00:23:38.192201 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":104,"threads_started":1}
I20260924 00:23:38.192680 14706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:23:38.192832 14703 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2aebaa49ce74c249f086f4d9e469185. Latest consensus state: current_term: 1 leader_uuid: "f2aebaa49ce74c249f086f4d9e469185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aebaa49ce74c249f086f4d9e469185" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } }
I20260924 00:23:38.193126 14703 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [sys.catalog]: This master's current role is: LEADER
I20260924 00:23:38.193224 14706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:23:38.194717 14706 catalog_manager.cc:1347] Generated new cluster ID: b3b48ff6ba7d493f985b08ae3c92029b
I20260924 00:23:38.194769 14706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:23:38.212508 14706 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:23:38.213104 14706 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:23:38.217758 14706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185: Generated new TSK 0
I20260924 00:23:38.218000 14706 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:23:38.252034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37889
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:23:38.337522 14713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:38.337790 14713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:38.337832 14713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:38.339061 14713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:38.339107 14713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:38.339134 14713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:23:38.340679 14713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:38.340936 14713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:38.341194 14713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:38.346843 14719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:38.346839 14721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:38.346877 14713 server_base.cc:1034] running on GCE node
W20260924 00:23:38.346835 14718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:38.347244 14713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:38.347486 14713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:38.348644 14713 hybrid_clock.cc:648] HybridClock initialized: now 1790209418348633 us; error 31 us; skew 500 ppm
Sep 24 00:23:38 dist-test-slave-777x krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:38.351284 14713 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:23:38.351701 14713 webserver.cc:460] Webserver started at http://127.0.98.1:37849/ using document root <none> and password file <none>
I20260924 00:23:38.351912 14713 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:38.351955 14713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:38.352046 14713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:23:38.353166 14713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "41ea6ed17b0c43af96c68ea8b238a5a3"
format_stamp: "Formatted at 2026-09-24 00:23:38 on dist-test-slave-777x"
server_key: "9a8971e8dd3892e4f49c7216b7cdec68"
server_key_iv: "3f00f90e86836ef438a4f6bc47c7e13e"
server_key_version: "encryptionkey@0"
I20260924 00:23:38.353479 14713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "41ea6ed17b0c43af96c68ea8b238a5a3"
format_stamp: "Formatted at 2026-09-24 00:23:38 on dist-test-slave-777x"
server_key: "9a8971e8dd3892e4f49c7216b7cdec68"
server_key_iv: "3f00f90e86836ef438a4f6bc47c7e13e"
server_key_version: "encryptionkey@0"
I20260924 00:23:38.354764 14713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:23:38.355461 14728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:38.355679 14713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:23:38.355772 14713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "41ea6ed17b0c43af96c68ea8b238a5a3"
format_stamp: "Formatted at 2026-09-24 00:23:38 on dist-test-slave-777x"
server_key: "9a8971e8dd3892e4f49c7216b7cdec68"
server_key_iv: "3f00f90e86836ef438a4f6bc47c7e13e"
server_key_version: "encryptionkey@0"
I20260924 00:23:38.355849 14713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:38.371613 14713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:38.374215 14713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:38.374383 14713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:38.374660 14713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:23:38.375007 14713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:23:38.375064 14713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:38.375109 14713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:23:38.375137 14713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:38.381592 14713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45893
I20260924 00:23:38.381634 14841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45893 every 8 connection(s)
I20260924 00:23:38.382009 14713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 00:23:38 dist-test-slave-777x krb5kdc[14605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:23:38.388725 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 14713
I20260924 00:23:38.388880 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:23:38.389092 392 external_mini_cluster.cc:1428] Setting key b0a35bc2f712b8cedeb6583c9de7c642
I20260924 00:23:38.393042 14844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:38.382783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.1:33685)
0924 00:23:38.382953 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:38.382997 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:38.383494 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:38.383745 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:38.383748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:38.384516 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:38.384727 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:38.384733 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.385530 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.385531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:38.386415 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:38.386418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.386801 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.386802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:38.386917 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:38.387501 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:38.387510 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:38.389108 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:38.391303 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:38.391314 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:38.391317 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:38.391742 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:38.391951 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:38.391952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:38.391952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:38.392126 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:38.392410 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:38.392418 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:38.392616 (+ 198us) client_negotiation.cc:769] Sending connection context
0924 00:23:38.392724 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 00:23:38.392865 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260924 00:23:38.393059 14689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:38.382831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33685 (local address 127.0.98.62:37889)
0924 00:23:38.382931 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:38.382933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:38.383109 (+ 176us) server_negotiation.cc:407] Connection header received
0924 00:23:38.383585 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:38.383586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:38.383597 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:38.383661 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:38.384862 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.385424 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.386510 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.386723 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.386781 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:38.389226 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:38.389227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:38.389228 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:38.389241 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:38.391190 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:38.391815 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:38.391816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:38.391816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:38.391873 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:38.392228 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:38.392229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:38.392229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:38.392342 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:38.392396 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:38.392789 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 00:23:38.392893 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 00:23:38.393755 14842 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37889
I20260924 00:23:38.393880 14842 heartbeater.cc:464] Registering TS with master...
I20260924 00:23:38.394189 14842 heartbeater.cc:511] Master 127.0.98.62:37889 requested a full tablet report, sending...
I20260924 00:23:38.394804 14650 ts_manager.cc:194] Registered new tserver with Master: 41ea6ed17b0c43af96c68ea8b238a5a3 (127.0.98.1:45893)
I20260924 00:23:38.395474 14650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33685
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:23:38.417110 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37889
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:23:38.501466 14847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:38.501694 14847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:38.501720 14847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:38.502894 14847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:38.502938 14847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:38.502965 14847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:23:38.504812 14847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:38.505052 14847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:38.505250 14847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:38.510612 14852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:38.510588 14853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:38.510591 14847 server_base.cc:1034] running on GCE node
W20260924 00:23:38.510588 14855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:38.511214 14847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:38.511454 14847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:38.512647 14847 hybrid_clock.cc:648] HybridClock initialized: now 1790209418512620 us; error 43 us; skew 500 ppm
Sep 24 00:23:38 dist-test-slave-777x krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:38.515385 14847 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:23:38.515899 14847 webserver.cc:460] Webserver started at http://127.0.98.2:38219/ using document root <none> and password file <none>
I20260924 00:23:38.516144 14847 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:38.516239 14847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:38.516398 14847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:23:38.517524 14847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "dec9aec545ff4260bd04ee91f1d4d581"
format_stamp: "Formatted at 2026-09-24 00:23:38 on dist-test-slave-777x"
server_key: "b88ead77189a15d78157e0a262c98d9b"
server_key_iv: "76101bc8239fbb284903c4c5e4c11c41"
server_key_version: "encryptionkey@0"
I20260924 00:23:38.517900 14847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dec9aec545ff4260bd04ee91f1d4d581"
format_stamp: "Formatted at 2026-09-24 00:23:38 on dist-test-slave-777x"
server_key: "b88ead77189a15d78157e0a262c98d9b"
server_key_iv: "76101bc8239fbb284903c4c5e4c11c41"
server_key_version: "encryptionkey@0"
I20260924 00:23:38.519310 14847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:23:38.520151 14862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:38.520434 14847 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:23:38.520546 14847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "dec9aec545ff4260bd04ee91f1d4d581"
format_stamp: "Formatted at 2026-09-24 00:23:38 on dist-test-slave-777x"
server_key: "b88ead77189a15d78157e0a262c98d9b"
server_key_iv: "76101bc8239fbb284903c4c5e4c11c41"
server_key_version: "encryptionkey@0"
I20260924 00:23:38.520635 14847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:38.537259 14847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:38.540253 14847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:38.540421 14847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:38.540675 14847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:23:38.540992 14847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:23:38.541033 14847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:38.541087 14847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:23:38.541115 14847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:38.547264 14847 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37727
I20260924 00:23:38.547340 14975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37727 every 8 connection(s)
I20260924 00:23:38.547672 14847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:23:38.552403 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 14847
I20260924 00:23:38.552531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:23:38.552769 392 external_mini_cluster.cc:1428] Setting key 92a4875d32b03ffdab7dca8848e3a7b1
Sep 24 00:23:38 dist-test-slave-777x krb5kdc[14605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:23:38.558421 14689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:38.548626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33551 (local address 127.0.98.62:37889)
0924 00:23:38.548694 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:38.548696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:38.548892 (+ 196us) server_negotiation.cc:407] Connection header received
0924 00:23:38.549348 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:38.549349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:38.549361 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:38.549430 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:38.550391 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.550907 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.551972 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.552153 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.552278 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:38.555300 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:38.555301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:38.555302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:38.555309 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:38.556770 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:38.557334 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:38.557335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:38.557336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:38.557403 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:38.557708 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:38.557709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:38.557709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:38.557820 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:38.557871 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:38.558244 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 00:23:38.558354 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 00:23:38.558758 14978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:38.548545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.2:33551)
0924 00:23:38.548733 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:38.548778 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:38.549263 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:38.549488 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:38.549492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:38.550093 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:38.550293 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:38.550299 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.551023 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.551024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:38.551872 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:38.551875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.552306 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.552308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:38.552569 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:38.553221 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:38.553232 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:38.555178 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:38.556869 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:38.556876 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:38.556879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:38.557242 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:38.557475 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:38.557477 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:38.557477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:38.557639 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:38.557887 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:38.557894 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:38.558086 (+ 192us) client_negotiation.cc:769] Sending connection context
0924 00:23:38.558220 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 00:23:38.558429 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":101,"threads_started":1}
I20260924 00:23:38.559345 14976 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37889
I20260924 00:23:38.559478 14976 heartbeater.cc:464] Registering TS with master...
I20260924 00:23:38.559726 14976 heartbeater.cc:511] Master 127.0.98.62:37889 requested a full tablet report, sending...
I20260924 00:23:38.560222 14650 ts_manager.cc:194] Registered new tserver with Master: dec9aec545ff4260bd04ee91f1d4d581 (127.0.98.2:37727)
I20260924 00:23:38.560618 14650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33551
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:23:38.587910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37889
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:23:38.675702 14981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:38.676100 14981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:38.676147 14981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:38.677672 14981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:38.677728 14981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:38.677760 14981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:23:38.679505 14981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:38.679792 14981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:38.680058 14981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:38.685773 14989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:38.685773 14986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:38.685834 14987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:38.686177 14981 server_base.cc:1034] running on GCE node
I20260924 00:23:38.686406 14981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:38.686621 14981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:38.687824 14981 hybrid_clock.cc:648] HybridClock initialized: now 1790209418687785 us; error 57 us; skew 500 ppm
Sep 24 00:23:38 dist-test-slave-777x krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:38.690828 14981 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:23:38.691283 14981 webserver.cc:460] Webserver started at http://127.0.98.3:45895/ using document root <none> and password file <none>
I20260924 00:23:38.691540 14981 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:38.691613 14981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:38.691772 14981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:23:38.693147 14981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "47ab9f4ab5794c09813b8c69f55b49b7"
format_stamp: "Formatted at 2026-09-24 00:23:38 on dist-test-slave-777x"
server_key: "11e113b9a001608a43d35e119f4dbbb8"
server_key_iv: "15380b16c0a289ce45150abaa8a97c98"
server_key_version: "encryptionkey@0"
I20260924 00:23:38.693538 14981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "47ab9f4ab5794c09813b8c69f55b49b7"
format_stamp: "Formatted at 2026-09-24 00:23:38 on dist-test-slave-777x"
server_key: "11e113b9a001608a43d35e119f4dbbb8"
server_key_iv: "15380b16c0a289ce45150abaa8a97c98"
server_key_version: "encryptionkey@0"
I20260924 00:23:38.694952 14981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:23:38.695833 14996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:38.696046 14981 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:23:38.696157 14981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "47ab9f4ab5794c09813b8c69f55b49b7"
format_stamp: "Formatted at 2026-09-24 00:23:38 on dist-test-slave-777x"
server_key: "11e113b9a001608a43d35e119f4dbbb8"
server_key_iv: "15380b16c0a289ce45150abaa8a97c98"
server_key_version: "encryptionkey@0"
I20260924 00:23:38.696310 14981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:38.721762 14981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:38.724432 14981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:38.724622 14981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:38.724897 14981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:23:38.725242 14981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:23:38.725301 14981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:38.725350 14981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:23:38.725375 14981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:38.731127 14981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38569
I20260924 00:23:38.731210 15109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38569 every 8 connection(s)
I20260924 00:23:38.731537 14981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:23:38.734015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 14981
I20260924 00:23:38.734125 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:23:38.734305 392 external_mini_cluster.cc:1428] Setting key 3bcb39938a2b4aa069f9743bb5679192
Sep 24 00:23:38 dist-test-slave-777x krb5kdc[14605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:23:38.742347 14689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:38.732388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47495 (local address 127.0.98.62:37889)
0924 00:23:38.732452 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:38.732454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:38.732672 (+ 218us) server_negotiation.cc:407] Connection header received
0924 00:23:38.733204 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:38.733205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:38.733217 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:38.733274 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:38.734484 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.735217 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.736504 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.736693 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.736747 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:38.739188 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:38.739190 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:38.739190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:38.739198 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:38.740811 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:38.741250 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:38.741251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:38.741252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:38.741318 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:38.741651 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:38.741652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:38.741652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:38.741783 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:38.741847 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:38.742201 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 00:23:38.742278 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 00:23:38.742465 15112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:38.732376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.3:47495)
0924 00:23:38.732541 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:38.732586 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:38.733096 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:38.733358 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:38.733362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:38.734042 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:38.734365 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:38.734372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.735328 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.735329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:38.736410 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:38.736413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.736765 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.736766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:38.736877 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:38.737553 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:38.737563 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:38.739056 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:38.740892 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:38.740899 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:38.740902 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:38.741153 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:38.741443 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:38.741444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:38.741444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:38.741573 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:38.741859 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:38.741865 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:38.742024 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 00:23:38.742130 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 00:23:38.742269 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260924 00:23:38.743095 15110 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37889
I20260924 00:23:38.743189 15110 heartbeater.cc:464] Registering TS with master...
I20260924 00:23:38.743379 15110 heartbeater.cc:511] Master 127.0.98.62:37889 requested a full tablet report, sending...
I20260924 00:23:38.743844 14650 ts_manager.cc:194] Registered new tserver with Master: 47ab9f4ab5794c09813b8c69f55b49b7 (127.0.98.3:38569)
I20260924 00:23:38.744297 14650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47495
I20260924 00:23:38.746345 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:23:38.754606 14689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:38.747779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50152 (local address 127.0.98.62:37889)
0924 00:23:38.747840 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:38.747842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:38.747860 (+ 18us) server_negotiation.cc:407] Connection header received
0924 00:23:38.747949 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:38.747950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:38.747959 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:38.748023 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:38.748635 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.749132 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.749869 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.750057 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.750118 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:38.751786 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:38.751787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:38.751788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:38.751796 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:38.753364 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:38.753748 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:38.753750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:38.753750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:38.753811 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:38.754018 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:38.754019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:38.754019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:38.754185 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:38.754321 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:38.754413 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 00:23:38.754499 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 00:23:38.756389 14650 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:23:38.757959 14650 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:23:38.765287 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:38.760817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45893 (local address 127.0.0.1:54866)
0924 00:23:38.761350 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:38.761362 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:38.761417 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:38.761717 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:38.761718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:38.761720 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:23:38.761771 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:38.761772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.763064 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.763065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:38.764054 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:38.764057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.764975 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.764976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:38.765062 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:38.765076 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 00:23:38.765144 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 00:23:38.765189 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":74,"threads_started":1}
I20260924 00:23:38.765327 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:38.760950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54866 (local address 127.0.98.1:45893)
0924 00:23:38.761124 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:38.761129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:38.761490 (+ 361us) server_negotiation.cc:407] Connection header received
0924 00:23:38.761539 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:38.761542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:38.761560 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:38.761637 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:23:38.761898 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.762945 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.764158 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.764880 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.764966 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:38.765030 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:38.765179 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 00:23:38.765224 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260924 00:23:38.765616 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:38.760584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37727 (local address 127.0.0.1:42856)
0924 00:23:38.760785 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:38.760833 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:38.760941 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:38.761225 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:38.761228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:38.761231 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:23:38.761393 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:38.761397 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.762865 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.762865 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:38.764378 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:38.764381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.765303 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.765304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:38.765373 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:38.765377 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:23:38.765438 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 00:23:38.765525 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":85,"thread_start_us":67,"threads_started":1}
I20260924 00:23:38.765964 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:38.760700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42856 (local address 127.0.98.2:37727)
0924 00:23:38.760920 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:38.760927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:38.760941 (+ 14us) server_negotiation.cc:407] Connection header received
0924 00:23:38.761044 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:38.761047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:38.761065 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:38.761200 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:23:38.761502 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.762752 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.764474 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.765208 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.765297 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:38.765356 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:38.765829 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 00:23:38.765874 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":101,"threads_started":1}
I20260924 00:23:38.766212 14776 tablet_service.cc:1467] Processing CreateTablet for tablet da6fa77c468c40d98384c258655e118b (DEFAULT_TABLE table=test-table [id=e4d6417d76004157a324adb825cf0665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:23:38.766634 14776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da6fa77c468c40d98384c258655e118b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:23:38.766650 14910 tablet_service.cc:1467] Processing CreateTablet for tablet da6fa77c468c40d98384c258655e118b (DEFAULT_TABLE table=test-table [id=e4d6417d76004157a324adb825cf0665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:23:38.767023 14910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da6fa77c468c40d98384c258655e118b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:23:38.769919 15127 tablet_bootstrap.cc:492] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3: Bootstrap starting.
I20260924 00:23:38.769958 15128 tablet_bootstrap.cc:492] T da6fa77c468c40d98384c258655e118b P dec9aec545ff4260bd04ee91f1d4d581: Bootstrap starting.
I20260924 00:23:38.770634 15128 tablet_bootstrap.cc:654] T da6fa77c468c40d98384c258655e118b P dec9aec545ff4260bd04ee91f1d4d581: Neither blocks nor log segments found. Creating new log.
I20260924 00:23:38.770648 15127 tablet_bootstrap.cc:654] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3: Neither blocks nor log segments found. Creating new log.
I20260924 00:23:38.771230 15127 log.cc:824] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:38.771241 15128 log.cc:824] T da6fa77c468c40d98384c258655e118b P dec9aec545ff4260bd04ee91f1d4d581: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:38.771561 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:38.760596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38569 (local address 127.0.0.1:47370)
0924 00:23:38.761151 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:38.761163 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:38.761223 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:38.761598 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:38.761599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:38.761602 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:23:38.761658 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:38.761660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.762756 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.762758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:38.764379 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:38.764381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.771208 (+ 6827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.771210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:38.771299 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:38.771306 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 00:23:38.771394 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 00:23:38.771462 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":37,"thread_start_us":130,"threads_started":1}
I20260924 00:23:38.771863 15126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:38.760969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47370 (local address 127.0.98.3:38569)
0924 00:23:38.761348 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:38.761354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:38.761365 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:23:38.761406 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:38.761408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:38.761425 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:38.761504 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:23:38.761763 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.762635 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.765400 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.771085 (+ 5685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.771192 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:38.771262 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:38.771704 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 00:23:38.771759 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":88,"threads_started":1}
I20260924 00:23:38.772365 15128 tablet_bootstrap.cc:492] T da6fa77c468c40d98384c258655e118b P dec9aec545ff4260bd04ee91f1d4d581: No bootstrap required, opened a new log
I20260924 00:23:38.772408 15127 tablet_bootstrap.cc:492] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3: No bootstrap required, opened a new log
I20260924 00:23:38.772455 15128 ts_tablet_manager.cc:1397] T da6fa77c468c40d98384c258655e118b P dec9aec545ff4260bd04ee91f1d4d581: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:23:38.772471 15127 ts_tablet_manager.cc:1397] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:23:38.772622 15044 tablet_service.cc:1467] Processing CreateTablet for tablet da6fa77c468c40d98384c258655e118b (DEFAULT_TABLE table=test-table [id=e4d6417d76004157a324adb825cf0665]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:23:38.773002 15044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da6fa77c468c40d98384c258655e118b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:23:38.774381 15128 raft_consensus.cc:348] T da6fa77c468c40d98384c258655e118b P dec9aec545ff4260bd04ee91f1d4d581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:23:38.774565 15128 raft_consensus.cc:374] T da6fa77c468c40d98384c258655e118b P dec9aec545ff4260bd04ee91f1d4d581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:23:38.774616 15128 raft_consensus.cc:729] T da6fa77c468c40d98384c258655e118b P dec9aec545ff4260bd04ee91f1d4d581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec9aec545ff4260bd04ee91f1d4d581, State: Initialized, Role: FOLLOWER
I20260924 00:23:38.774770 15128 consensus_queue.cc:260] T da6fa77c468c40d98384c258655e118b P dec9aec545ff4260bd04ee91f1d4d581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:23:38.774660 15127 raft_consensus.cc:348] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:23:38.774863 15127 raft_consensus.cc:374] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:23:38.774914 15127 raft_consensus.cc:729] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41ea6ed17b0c43af96c68ea8b238a5a3, State: Initialized, Role: FOLLOWER
I20260924 00:23:38.775106 15128 ts_tablet_manager.cc:1428] T da6fa77c468c40d98384c258655e118b P dec9aec545ff4260bd04ee91f1d4d581: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:23:38.775084 15127 consensus_queue.cc:260] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:23:38.775266 14976 heartbeater.cc:503] Master 127.0.98.62:37889 was elected leader, sending a full tablet report...
I20260924 00:23:38.775390 15127 ts_tablet_manager.cc:1428] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:23:38.775413 14842 heartbeater.cc:503] Master 127.0.98.62:37889 was elected leader, sending a full tablet report...
I20260924 00:23:38.776746 15133 tablet_bootstrap.cc:492] T da6fa77c468c40d98384c258655e118b P 47ab9f4ab5794c09813b8c69f55b49b7: Bootstrap starting.
I20260924 00:23:38.777523 15133 tablet_bootstrap.cc:654] T da6fa77c468c40d98384c258655e118b P 47ab9f4ab5794c09813b8c69f55b49b7: Neither blocks nor log segments found. Creating new log.
I20260924 00:23:38.778023 15133 log.cc:824] T da6fa77c468c40d98384c258655e118b P 47ab9f4ab5794c09813b8c69f55b49b7: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:38.778774 15133 tablet_bootstrap.cc:492] T da6fa77c468c40d98384c258655e118b P 47ab9f4ab5794c09813b8c69f55b49b7: No bootstrap required, opened a new log
I20260924 00:23:38.778861 15133 ts_tablet_manager.cc:1397] T da6fa77c468c40d98384c258655e118b P 47ab9f4ab5794c09813b8c69f55b49b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:23:38.780452 15133 raft_consensus.cc:348] T da6fa77c468c40d98384c258655e118b P 47ab9f4ab5794c09813b8c69f55b49b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:23:38.780623 15133 raft_consensus.cc:374] T da6fa77c468c40d98384c258655e118b P 47ab9f4ab5794c09813b8c69f55b49b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:23:38.780674 15133 raft_consensus.cc:729] T da6fa77c468c40d98384c258655e118b P 47ab9f4ab5794c09813b8c69f55b49b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47ab9f4ab5794c09813b8c69f55b49b7, State: Initialized, Role: FOLLOWER
I20260924 00:23:38.780815 15133 consensus_queue.cc:260] T da6fa77c468c40d98384c258655e118b P 47ab9f4ab5794c09813b8c69f55b49b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:23:38.781095 15133 ts_tablet_manager.cc:1428] T da6fa77c468c40d98384c258655e118b P 47ab9f4ab5794c09813b8c69f55b49b7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 00:23:38.781129 15110 heartbeater.cc:503] Master 127.0.98.62:37889 was elected leader, sending a full tablet report...
W20260924 00:23:38.798354 14977 tablet.cc:2369] T da6fa77c468c40d98384c258655e118b P dec9aec545ff4260bd04ee91f1d4d581: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:23:38.857553 15132 raft_consensus.cc:482] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:38.857687 15132 raft_consensus.cc:504] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:23:38.858175 15132 leader_election.cc:288] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47ab9f4ab5794c09813b8c69f55b49b7 (127.0.98.3:38569), dec9aec545ff4260bd04ee91f1d4d581 (127.0.98.2:37727)
I20260924 00:23:38.861621 15126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:38.858527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60183 (local address 127.0.98.3:38569)
0924 00:23:38.858585 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:38.858587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:38.858600 (+ 13us) server_negotiation.cc:407] Connection header received
0924 00:23:38.858643 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:38.858644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:38.858655 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:38.858709 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:23:38.858921 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.859603 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.860711 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.861236 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.861337 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:38.861350 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:38.861503 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 00:23:38.861544 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 00:23:38.861621 14844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:38.858361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38569 (local address 127.0.98.1:60183)
0924 00:23:38.858453 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:38.858469 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:38.858535 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:38.858727 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:38.858728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:38.858730 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:23:38.858802 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:38.858804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.859686 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.859687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:38.860608 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:38.860610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.861331 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.861332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:38.861422 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:38.861426 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:23:38.861495 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 00:23:38.861544 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 00:23:38.861809 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:38.858407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37727 (local address 127.0.98.1:41519)
0924 00:23:38.858623 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:38.858633 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:38.858684 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:38.858865 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:38.858865 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:38.858867 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:23:38.858949 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:38.858950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.859760 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.859761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:38.860699 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:38.860701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.861483 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.861484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:38.861563 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:38.861567 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:23:38.861647 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 00:23:38.861692 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":99,"threads_started":1}
I20260924 00:23:38.862062 15064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6fa77c468c40d98384c258655e118b" candidate_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" is_pre_election: true
I20260924 00:23:38.862191 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:38.858614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41519 (local address 127.0.98.2:37727)
0924 00:23:38.858680 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:38.858682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:38.858692 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:23:38.858740 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:38.858741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:38.858753 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:38.858862 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:23:38.859040 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.859661 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.860819 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.861392 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.861483 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:38.861493 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:38.861657 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 00:23:38.862129 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 00:23:38.862258 15064 raft_consensus.cc:2454] T da6fa77c468c40d98384c258655e118b P 47ab9f4ab5794c09813b8c69f55b49b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41ea6ed17b0c43af96c68ea8b238a5a3 in term 0.
I20260924 00:23:38.862473 14930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6fa77c468c40d98384c258655e118b" candidate_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec9aec545ff4260bd04ee91f1d4d581" is_pre_election: true
I20260924 00:23:38.862488 14730 leader_election.cc:302] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41ea6ed17b0c43af96c68ea8b238a5a3, 47ab9f4ab5794c09813b8c69f55b49b7; no voters:
I20260924 00:23:38.862591 14930 raft_consensus.cc:2454] T da6fa77c468c40d98384c258655e118b P dec9aec545ff4260bd04ee91f1d4d581 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41ea6ed17b0c43af96c68ea8b238a5a3 in term 0.
I20260924 00:23:38.862643 15132 raft_consensus.cc:2781] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:23:38.862699 15132 raft_consensus.cc:482] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:38.862727 15132 raft_consensus.cc:3037] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:23:38.863466 15132 raft_consensus.cc:504] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:23:38.863626 15132 leader_election.cc:288] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 1 election: Requested vote from peers 47ab9f4ab5794c09813b8c69f55b49b7 (127.0.98.3:38569), dec9aec545ff4260bd04ee91f1d4d581 (127.0.98.2:37727)
I20260924 00:23:38.863754 15064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6fa77c468c40d98384c258655e118b" candidate_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47ab9f4ab5794c09813b8c69f55b49b7"
I20260924 00:23:38.863837 15064 raft_consensus.cc:3037] T da6fa77c468c40d98384c258655e118b P 47ab9f4ab5794c09813b8c69f55b49b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:23:38.863891 14930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6fa77c468c40d98384c258655e118b" candidate_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec9aec545ff4260bd04ee91f1d4d581"
I20260924 00:23:38.863957 14930 raft_consensus.cc:3037] T da6fa77c468c40d98384c258655e118b P dec9aec545ff4260bd04ee91f1d4d581 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:23:38.864583 15064 raft_consensus.cc:2454] T da6fa77c468c40d98384c258655e118b P 47ab9f4ab5794c09813b8c69f55b49b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41ea6ed17b0c43af96c68ea8b238a5a3 in term 1.
I20260924 00:23:38.864776 14730 leader_election.cc:302] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41ea6ed17b0c43af96c68ea8b238a5a3, 47ab9f4ab5794c09813b8c69f55b49b7; no voters:
I20260924 00:23:38.864799 14930 raft_consensus.cc:2454] T da6fa77c468c40d98384c258655e118b P dec9aec545ff4260bd04ee91f1d4d581 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41ea6ed17b0c43af96c68ea8b238a5a3 in term 1.
I20260924 00:23:38.864863 15132 raft_consensus.cc:2781] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:23:38.865028 15132 raft_consensus.cc:686] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 LEADER]: Becoming Leader. State: Replica: 41ea6ed17b0c43af96c68ea8b238a5a3, State: Running, Role: LEADER
I20260924 00:23:38.865154 15132 consensus_queue.cc:237] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:23:38.866019 14649 catalog_manager.cc:5495] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41ea6ed17b0c43af96c68ea8b238a5a3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } health_report { overall_health: HEALTHY } } }
I20260924 00:23:38.882372 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:38.879694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54878 (local address 127.0.98.1:45893)
0924 00:23:38.879741 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:38.879743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:38.879751 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:23:38.879795 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:38.879796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:38.879824 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:38.879920 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:23:38.880136 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.880703 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.881491 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:38.881700 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:38.881793 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:38.881986 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:23:38.882119 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:23:38.882218 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:38.882237 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 00:23:38.882258 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 00:23:38.882766 14843 tablet.cc:2369] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:23:38.884302 15064 raft_consensus.cc:1261] T da6fa77c468c40d98384c258655e118b P 47ab9f4ab5794c09813b8c69f55b49b7 [term 1 FOLLOWER]: Refusing update from remote peer 41ea6ed17b0c43af96c68ea8b238a5a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:23:38.884330 14930 raft_consensus.cc:1261] T da6fa77c468c40d98384c258655e118b P dec9aec545ff4260bd04ee91f1d4d581 [term 1 FOLLOWER]: Refusing update from remote peer 41ea6ed17b0c43af96c68ea8b238a5a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:23:38.884657 15132 consensus_queue.cc:1035] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:23:38.884845 15137 consensus_queue.cc:1035] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:23:38.887373 15140 mvcc.cc:205] Tried to move back new op lower bound from 7332697779747172352 to 7332697779672203264. Current Snapshot: MvccSnapshot[applied={T|T < 7332697779747172352}]
I20260924 00:23:38.887545 15141 mvcc.cc:205] Tried to move back new op lower bound from 7332697779747172352 to 7332697779672203264. Current Snapshot: MvccSnapshot[applied={T|T < 7332697779747172352}]
I20260924 00:23:38.887769 15142 mvcc.cc:205] Tried to move back new op lower bound from 7332697779747172352 to 7332697779672203264. Current Snapshot: MvccSnapshot[applied={T|T < 7332697779747172352}]
I20260924 00:23:38.890412 14650 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:23:38.890545 14650 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50152:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:23:38.891378 14650 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185: Sending DeleteTablet for 3 replicas of tablet da6fa77c468c40d98384c258655e118b
I20260924 00:23:38.891772 14776 tablet_service.cc:1514] Processing DeleteTablet for tablet da6fa77c468c40d98384c258655e118b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:23:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54866
I20260924 00:23:38.891796 15044 tablet_service.cc:1514] Processing DeleteTablet for tablet da6fa77c468c40d98384c258655e118b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:23:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47370
I20260924 00:23:38.892086 14910 tablet_service.cc:1514] Processing DeleteTablet for tablet da6fa77c468c40d98384c258655e118b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:23:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42856
I20260924 00:23:38.892124 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 14713
I20260924 00:23:38.892793 15152 tablet_replica.cc:330] T da6fa77c468c40d98384c258655e118b P dec9aec545ff4260bd04ee91f1d4d581: stopping tablet replica
I20260924 00:23:38.892953 15152 raft_consensus.cc:2229] T da6fa77c468c40d98384c258655e118b P dec9aec545ff4260bd04ee91f1d4d581 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:23:38.893131 15152 raft_consensus.cc:2258] T da6fa77c468c40d98384c258655e118b P dec9aec545ff4260bd04ee91f1d4d581 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:23:38.893235 15151 tablet_replica.cc:330] T da6fa77c468c40d98384c258655e118b P 47ab9f4ab5794c09813b8c69f55b49b7: stopping tablet replica
I20260924 00:23:38.893383 15151 raft_consensus.cc:2229] T da6fa77c468c40d98384c258655e118b P 47ab9f4ab5794c09813b8c69f55b49b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:23:38.893496 15151 raft_consensus.cc:2258] T da6fa77c468c40d98384c258655e118b P 47ab9f4ab5794c09813b8c69f55b49b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:23:38.893781 15152 ts_tablet_manager.cc:1905] T da6fa77c468c40d98384c258655e118b P dec9aec545ff4260bd04ee91f1d4d581: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:23:38.893991 15151 ts_tablet_manager.cc:1905] T da6fa77c468c40d98384c258655e118b P 47ab9f4ab5794c09813b8c69f55b49b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:23:38.895637 15152 ts_tablet_manager.cc:1918] T da6fa77c468c40d98384c258655e118b P dec9aec545ff4260bd04ee91f1d4d581: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:23:38.895717 15152 log.cc:1196] T da6fa77c468c40d98384c258655e118b P dec9aec545ff4260bd04ee91f1d4d581: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/wal/wals/da6fa77c468c40d98384c258655e118b
I20260924 00:23:38.895673 15151 ts_tablet_manager.cc:1918] T da6fa77c468c40d98384c258655e118b P 47ab9f4ab5794c09813b8c69f55b49b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:23:38.895787 15151 log.cc:1196] T da6fa77c468c40d98384c258655e118b P 47ab9f4ab5794c09813b8c69f55b49b7: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/da6fa77c468c40d98384c258655e118b
I20260924 00:23:38.895989 15152 ts_tablet_manager.cc:1939] T da6fa77c468c40d98384c258655e118b P dec9aec545ff4260bd04ee91f1d4d581: Deleting consensus metadata
I20260924 00:23:38.896142 15151 ts_tablet_manager.cc:1939] T da6fa77c468c40d98384c258655e118b P 47ab9f4ab5794c09813b8c69f55b49b7: Deleting consensus metadata
I20260924 00:23:38.896472 14637 catalog_manager.cc:4841] TS dec9aec545ff4260bd04ee91f1d4d581 (127.0.98.2:37727): tablet da6fa77c468c40d98384c258655e118b (table test-table [id=e4d6417d76004157a324adb825cf0665]) successfully deleted
I20260924 00:23:38.896674 14635 catalog_manager.cc:4841] TS 47ab9f4ab5794c09813b8c69f55b49b7 (127.0.98.3:38569): tablet da6fa77c468c40d98384c258655e118b (table test-table [id=e4d6417d76004157a324adb825cf0665]) successfully deleted
W20260924 00:23:38.898485 14637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45893 (error 108)
I20260924 00:23:38.898746 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 14847
I20260924 00:23:38.903977 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:38.903816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45893 (local address 127.0.0.1:54886)
0924 00:23:38.903876 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:38.903929 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 00:23:38.904084 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 14981
W20260924 00:23:38.904191 14637 catalog_manager.cc:4584] TS 41ea6ed17b0c43af96c68ea8b238a5a3 (127.0.98.1:45893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da6fa77c468c40d98384c258655e118b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45893: connect: Connection refused (error 111)
I20260924 00:23:38.908812 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 14618
I20260924 00:23:58.914693 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37889
--webserver_interface=127.0.98.62
--webserver_port=39923
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:23:58.997889 15154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:58.998126 15154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:58.998150 15154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:58.999390 15154 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 00:23:58.999436 15154 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 00:23:58.999450 15154 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 00:23:58.999462 15154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:23:58.999475 15154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:58.999490 15154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:58.999501 15154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:23:58.999513 15154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:23:59.001202 15154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:59.001488 15154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:59.001732 15154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:59.010901 15163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:59.010982 15160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:59.011016 15161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:59.011003 15154 server_base.cc:1034] running on GCE node
I20260924 00:23:59.011445 15154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:59.011771 15154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:59.012960 15154 hybrid_clock.cc:648] HybridClock initialized: now 1790209439012955 us; error 55 us; skew 500 ppm
Sep 24 00:23:59 dist-test-slave-777x krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:59.015961 15154 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:23:59.016531 15154 webserver.cc:460] Webserver started at http://127.0.98.62:39923/ using document root <none> and password file <none>
I20260924 00:23:59.016800 15154 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:59.016898 15154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:59.018496 15154 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:23:59.019246 15170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:59.019517 15154 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:23:59.019732 15154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "f2aebaa49ce74c249f086f4d9e469185"
format_stamp: "Formatted at 2026-09-24 00:23:38 on dist-test-slave-777x"
server_key: "ecb3874a00faf02d94c6b15f69a48452"
server_key_iv: "de57ad7cc71d808701a4d47ac514f40e"
server_key_version: "encryptionkey@0"
I20260924 00:23:59.020303 15154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:59.029100 15154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:59.031679 15154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:59.031857 15154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:59.036299 15154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37889
I20260924 00:23:59.036376 15222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37889 every 8 connection(s)
I20260924 00:23:59.036782 15154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:23:59.038460 15223 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:23:59.039079 15223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185: Bootstrap starting.
I20260924 00:23:59.040387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 15154
I20260924 00:23:59.040675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45893
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37889
--webserver_port=37849
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:23:59.049926 15223 log.cc:824] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:59.053076 15223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:23:59.053478 15223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185: Bootstrap complete.
I20260924 00:23:59.056805 15223 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aebaa49ce74c249f086f4d9e469185" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } }
I20260924 00:23:59.057201 15223 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2aebaa49ce74c249f086f4d9e469185, State: Initialized, Role: FOLLOWER
I20260924 00:23:59.057370 15223 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aebaa49ce74c249f086f4d9e469185" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } }
I20260924 00:23:59.057436 15223 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:23:59.057477 15223 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:23:59.057524 15223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 1 FOLLOWER]: Advancing to term 2
I20260924 00:23:59.058921 15223 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aebaa49ce74c249f086f4d9e469185" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } }
I20260924 00:23:59.059036 15223 leader_election.cc:302] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2aebaa49ce74c249f086f4d9e469185; no voters:
I20260924 00:23:59.059211 15223 leader_election.cc:288] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 00:23:59.059294 15227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 00:23:59.059460 15223 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:23:59.059504 15227 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 2 LEADER]: Becoming Leader. State: Replica: f2aebaa49ce74c249f086f4d9e469185, State: Running, Role: LEADER
I20260924 00:23:59.059648 15227 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aebaa49ce74c249f086f4d9e469185" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } }
I20260924 00:23:59.060017 15229 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2aebaa49ce74c249f086f4d9e469185. Latest consensus state: current_term: 2 leader_uuid: "f2aebaa49ce74c249f086f4d9e469185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aebaa49ce74c249f086f4d9e469185" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } }
I20260924 00:23:59.060271 15229 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [sys.catalog]: This master's current role is: LEADER
I20260924 00:23:59.060721 15228 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f2aebaa49ce74c249f086f4d9e469185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aebaa49ce74c249f086f4d9e469185" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } }
I20260924 00:23:59.060873 15228 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [sys.catalog]: This master's current role is: LEADER
I20260924 00:23:59.061036 15238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:23:59.061869 15238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:23:59.062031 15238 catalog_manager.cc:1259] Loaded cluster ID: b3b48ff6ba7d493f985b08ae3c92029b
I20260924 00:23:59.062269 15238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:23:59.063097 15238 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:23:59.063230 15238 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185: Loaded TSK: 0
I20260924 00:23:59.071264 15238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185: Generated new TSK 1
I20260924 00:23:59.071396 15238 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 00:23:59.145493 15225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:59.145737 15225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:59.145766 15225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:59.147403 15225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:59.147464 15225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:59.147495 15225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:23:59.149408 15225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37849
--tserver_master_addrs=127.0.98.62:37889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:59.149698 15225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:59.150012 15225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:59.155681 15250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:59.155949 15225 server_base.cc:1034] running on GCE node
W20260924 00:23:59.155738 15253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:59.155681 15251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:59.156312 15225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:59.156661 15225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:59.157972 15225 hybrid_clock.cc:648] HybridClock initialized: now 1790209439157941 us; error 47 us; skew 500 ppm
Sep 24 00:23:59 dist-test-slave-777x krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:59.160920 15225 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:23:59.161540 15225 webserver.cc:460] Webserver started at http://127.0.98.1:37849/ using document root <none> and password file <none>
I20260924 00:23:59.161803 15225 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:59.161886 15225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:59.163499 15225 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:23:59.164451 15260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:59.164667 15225 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:23:59.164798 15225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "41ea6ed17b0c43af96c68ea8b238a5a3"
format_stamp: "Formatted at 2026-09-24 00:23:38 on dist-test-slave-777x"
server_key: "9a8971e8dd3892e4f49c7216b7cdec68"
server_key_iv: "3f00f90e86836ef438a4f6bc47c7e13e"
server_key_version: "encryptionkey@0"
I20260924 00:23:59.165238 15225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:59.182710 15225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:59.185451 15225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:59.185639 15225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:59.185905 15225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:23:59.186676 15267 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 00:23:59.188736 15225 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 00:23:59.188819 15225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 00:23:59.188876 15225 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 00:23:59.189597 15225 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 00:23:59.189657 15225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 00:23:59.189733 15267 tablet_bootstrap.cc:492] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3: Bootstrap starting.
I20260924 00:23:59.197935 15225 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45893
I20260924 00:23:59.197988 15374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45893 every 8 connection(s)
I20260924 00:23:59.198464 15225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:23:59.200882 15267 log.cc:824] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:59.203634 15267 tablet_bootstrap.cc:492] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:23:59.204005 15267 tablet_bootstrap.cc:492] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3: Bootstrap complete.
I20260924 00:23:59.205091 15267 ts_tablet_manager.cc:1397] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.013s
I20260924 00:23:59.206643 15267 raft_consensus.cc:348] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:23:59.207001 15267 raft_consensus.cc:729] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41ea6ed17b0c43af96c68ea8b238a5a3, State: Initialized, Role: FOLLOWER
I20260924 00:23:59.207201 15267 consensus_queue.cc:260] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:23:59.207391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 15225
I20260924 00:23:59.207562 15267 ts_tablet_manager.cc:1428] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20260924 00:23:59.207738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37727
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37889
--webserver_port=38219
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 00:23:59 dist-test-slave-777x krb5kdc[14605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:23:59.215014 15378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:59.199342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54315 (local address 127.0.98.62:37889)
0924 00:23:59.202728 (+ 3386us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:59.202739 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:59.202780 (+ 41us) server_negotiation.cc:407] Connection header received
0924 00:23:59.202916 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:59.202923 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:59.202978 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:59.203120 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:59.204084 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.205485 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.207083 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.207812 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.207954 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:59.210734 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:59.210739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:59.210739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:59.210750 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:59.212725 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:59.213367 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:59.213370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:59.213370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:59.213486 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:59.213874 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:59.213876 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:59.213876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:59.214146 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:59.214274 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:59.214675 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 00:23:59.214848 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3328,"thread_start_us":79,"threads_started":1}
I20260924 00:23:59.215158 15377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:59.199263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.1:54315)
0924 00:23:59.201296 (+ 2033us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:59.201359 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:59.202104 (+ 745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:59.203200 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:59.203205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:59.203622 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:59.203935 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:59.203943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.205607 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.205608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:59.206955 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:59.206958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.207994 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.207995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:59.208138 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:59.208903 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:59.208913 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:59.210599 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:59.212830 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:59.212840 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:59.212844 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:59.213244 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:59.213567 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:59.213568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:59.213568 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:59.213757 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:59.214244 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:59.214252 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:59.214461 (+ 209us) client_negotiation.cc:769] Sending connection context
0924 00:23:59.214609 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 00:23:59.214857 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1953,"thread_start_us":95,"threads_started":1}
I20260924 00:23:59.216010 15375 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37889
I20260924 00:23:59.216144 15375 heartbeater.cc:464] Registering TS with master...
I20260924 00:23:59.216410 15375 heartbeater.cc:511] Master 127.0.98.62:37889 requested a full tablet report, sending...
I20260924 00:23:59.217237 15187 ts_manager.cc:194] Registered new tserver with Master: 41ea6ed17b0c43af96c68ea8b238a5a3 (127.0.98.1:45893)
I20260924 00:23:59.218369 15187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54315
I20260924 00:23:59.219449 15375 heartbeater.cc:503] Master 127.0.98.62:37889 was elected leader, sending a full tablet report...
Sep 24 00:23:59 dist-test-slave-777x krb5kdc[14605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 00:23:59.227977 15382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:59.217910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45893 (local address 127.0.0.1:35022)
0924 00:23:59.218154 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:59.218209 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:59.218302 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:59.219077 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:59.219086 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:59.219545 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:59.219625 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:59.219627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.220845 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.220847 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:59.221916 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:59.221919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.222443 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.222447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:59.222547 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:59.223276 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:59.223281 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:59.224779 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:59.226696 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:59.226704 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:59.226706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:59.226928 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:59.227171 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:59.227172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:59.227172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:59.227238 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:59.227559 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:59.227566 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:59.227621 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 00:23:59.227723 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 00:23:59.227864 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":116,"threads_started":1}
I20260924 00:23:59.227988 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:59.218707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.0.98.1:45893)
0924 00:23:59.218893 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:59.218898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:59.218909 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:23:59.218946 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:59.218948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:59.218965 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:59.219050 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:59.219904 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.220758 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.222026 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.222357 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.222425 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:59.224886 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:59.224902 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:59.224902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:59.224916 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:59.226569 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:59.227030 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:59.227033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:59.227034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:59.227097 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:59.227303 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:59.227304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:59.227304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:59.227476 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:59.227568 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:59.227770 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 00:23:59.227866 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":119,"threads_started":1}
I20260924 00:23:59.228549 15292 tablet_service.cc:1514] Processing DeleteTablet for tablet da6fa77c468c40d98384c258655e118b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:23:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35022
I20260924 00:23:59.229211 15384 tablet_replica.cc:330] stopping tablet replica
I20260924 00:23:59.229375 15384 raft_consensus.cc:2229] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:23:59.229482 15384 raft_consensus.cc:2258] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:23:59.229797 15384 ts_tablet_manager.cc:1905] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:23:59.232299 15384 ts_tablet_manager.cc:1918] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:23:59.232400 15384 log.cc:1196] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/wal/wals/da6fa77c468c40d98384c258655e118b
I20260924 00:23:59.232658 15384 ts_tablet_manager.cc:1939] T da6fa77c468c40d98384c258655e118b P 41ea6ed17b0c43af96c68ea8b238a5a3: Deleting consensus metadata
I20260924 00:23:59.233235 15174 catalog_manager.cc:4841] TS 41ea6ed17b0c43af96c68ea8b238a5a3 (127.0.98.1:45893): tablet da6fa77c468c40d98384c258655e118b (table test-table [id=e4d6417d76004157a324adb825cf0665]) successfully deleted
W20260924 00:23:59.305722 15381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:59.306032 15381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:59.306082 15381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:59.307456 15381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:59.307538 15381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:59.307590 15381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:23:59.309303 15381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38219
--tserver_master_addrs=127.0.98.62:37889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:59.309625 15381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:59.309899 15381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:59.316188 15390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:59.316203 15389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:59.316207 15392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:59.316515 15381 server_base.cc:1034] running on GCE node
I20260924 00:23:59.316810 15381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:59.317090 15381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:59.318307 15381 hybrid_clock.cc:648] HybridClock initialized: now 1790209439318287 us; error 56 us; skew 500 ppm
Sep 24 00:23:59 dist-test-slave-777x krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:59.321130 15381 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:23:59.321657 15381 webserver.cc:460] Webserver started at http://127.0.98.2:38219/ using document root <none> and password file <none>
I20260924 00:23:59.321911 15381 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:59.322006 15381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:59.323554 15381 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:23:59.324288 15399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:59.324481 15381 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:23:59.324575 15381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "dec9aec545ff4260bd04ee91f1d4d581"
format_stamp: "Formatted at 2026-09-24 00:23:38 on dist-test-slave-777x"
server_key: "b88ead77189a15d78157e0a262c98d9b"
server_key_iv: "76101bc8239fbb284903c4c5e4c11c41"
server_key_version: "encryptionkey@0"
I20260924 00:23:59.325001 15381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:59.347513 15381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:59.350419 15381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:59.350636 15381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:59.350935 15381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:23:59.351289 15381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:23:59.351351 15381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:59.351400 15381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:23:59.351429 15381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:59.358543 15381 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37727
I20260924 00:23:59.358587 15512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37727 every 8 connection(s)
I20260924 00:23:59.358976 15381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:23:59.364112 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 15381
I20260924 00:23:59.364504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38569
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37889
--webserver_port=45895
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 00:23:59 dist-test-slave-777x krb5kdc[14605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:23:59.369732 15378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:59.359795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59713 (local address 127.0.98.62:37889)
0924 00:23:59.359918 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:59.359921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:59.360123 (+ 202us) server_negotiation.cc:407] Connection header received
0924 00:23:59.360730 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:59.360731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:59.360747 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:59.360811 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:59.362005 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.362551 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.363664 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.363911 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.364010 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:59.366579 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:59.366582 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:59.366582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:59.366591 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:59.368208 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:59.368626 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:59.368628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:59.368628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:59.368735 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:59.369042 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:59.369043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:59.369044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:59.369170 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:59.369241 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:59.369544 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 00:23:59.369640 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 00:23:59.370011 15515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:59.359795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.2:59713)
0924 00:23:59.359991 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:59.360030 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:59.360607 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:59.360897 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:59.360901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:59.361615 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:59.361892 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:59.361898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.362682 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.362683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:59.363553 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:59.363556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.364024 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.364025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:59.364224 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:59.364900 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:59.364908 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:59.366468 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:59.368284 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:59.368292 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:59.368294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:59.368543 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:59.368820 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:59.368822 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:59.368822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:59.368963 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:59.369244 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:59.369252 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:59.369402 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 00:23:59.369512 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 00:23:59.369687 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":111,"threads_started":1}
I20260924 00:23:59.370575 15513 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37889
I20260924 00:23:59.370677 15513 heartbeater.cc:464] Registering TS with master...
I20260924 00:23:59.370888 15513 heartbeater.cc:511] Master 127.0.98.62:37889 requested a full tablet report, sending...
I20260924 00:23:59.371335 15187 ts_manager.cc:194] Registered new tserver with Master: dec9aec545ff4260bd04ee91f1d4d581 (127.0.98.2:37727)
I20260924 00:23:59.371973 15187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59713
W20260924 00:23:59.452934 15516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:23:59.453132 15516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:23:59.453155 15516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:23:59.454236 15516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:23:59.454270 15516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:23:59.454298 15516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:23:59.455680 15516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45895
--tserver_master_addrs=127.0.98.62:37889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:23:59.455878 15516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:23:59.456050 15516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:23:59.461316 15521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:23:59.461398 15524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:59.461562 15516 server_base.cc:1034] running on GCE node
W20260924 00:23:59.461688 15522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:23:59.461915 15516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:23:59.462139 15516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:23:59.463340 15516 hybrid_clock.cc:648] HybridClock initialized: now 1790209439463320 us; error 34 us; skew 500 ppm
Sep 24 00:23:59 dist-test-slave-777x krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:23:59.466337 15516 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:23:59.466890 15516 webserver.cc:460] Webserver started at http://127.0.98.3:45895/ using document root <none> and password file <none>
I20260924 00:23:59.467137 15516 fs_manager.cc:359] Metadata directory not provided
I20260924 00:23:59.467228 15516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:23:59.468747 15516 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 00:23:59.469516 15531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:59.469692 15516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:23:59.469797 15516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "47ab9f4ab5794c09813b8c69f55b49b7"
format_stamp: "Formatted at 2026-09-24 00:23:38 on dist-test-slave-777x"
server_key: "11e113b9a001608a43d35e119f4dbbb8"
server_key_iv: "15380b16c0a289ce45150abaa8a97c98"
server_key_version: "encryptionkey@0"
I20260924 00:23:59.470149 15516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:23:59.482400 15516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:23:59.484753 15516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:23:59.484956 15516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:23:59.485216 15516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:23:59.485528 15516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:23:59.485584 15516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:59.485625 15516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:23:59.485652 15516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:23:59.492069 15516 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38569
I20260924 00:23:59.492117 15644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38569 every 8 connection(s)
I20260924 00:23:59.492487 15516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 00:23:59 dist-test-slave-777x krb5kdc[14605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:23:59.500041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 15516
I20260924 00:23:59.503237 15378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:59.493198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35197 (local address 127.0.98.62:37889)
0924 00:23:59.493270 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:59.493272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:59.493491 (+ 219us) server_negotiation.cc:407] Connection header received
0924 00:23:59.494009 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:59.494010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:59.494020 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:59.494082 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:59.495057 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.495628 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.496794 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.496999 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.497071 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:59.499601 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:59.499603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:59.499603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:59.499612 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:59.501449 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:59.502052 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:59.502053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:59.502053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:59.502111 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:59.502454 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:59.502455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:59.502455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:59.502594 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:59.502648 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:59.503056 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 00:23:59.503142 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 00:23:59.503420 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:59.493190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.3:35197)
0924 00:23:59.493342 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:59.493384 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:59.493910 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:59.494137 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:59.494141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:59.494712 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:23:59.494914 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:59.494919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.495731 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.495732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:59.496701 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:59.496704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.497104 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.497105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:59.497223 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:59.497813 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:23:59.497822 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:23:59.499486 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:23:59.501553 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:59.501567 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:59.501569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:59.501939 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:59.502191 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:23:59.502192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:23:59.502192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:23:59.502374 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:23:59.502652 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:23:59.502658 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:23:59.502870 (+ 212us) client_negotiation.cc:769] Sending connection context
0924 00:23:59.503003 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 00:23:59.503177 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260924 00:23:59.504016 15645 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37889
I20260924 00:23:59.504146 15645 heartbeater.cc:464] Registering TS with master...
I20260924 00:23:59.504384 15645 heartbeater.cc:511] Master 127.0.98.62:37889 requested a full tablet report, sending...
I20260924 00:23:59.504870 15187 ts_manager.cc:194] Registered new tserver with Master: 47ab9f4ab5794c09813b8c69f55b49b7 (127.0.98.3:38569)
I20260924 00:23:59.505308 15187 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35197
I20260924 00:23:59.508719 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:59.500521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37080 (local address 127.0.98.62:37889)
0924 00:23:59.500658 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:59.500660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:59.500785 (+ 125us) server_negotiation.cc:407] Connection header received
0924 00:23:59.500859 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:59.500913 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:59.500926 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:59.500986 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:59.501750 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.502894 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.504106 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.504405 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.504482 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:59.505970 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:59.505971 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:59.505972 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:59.505979 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:59.507479 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:59.507840 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:59.507841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:59.507842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:59.507899 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:59.508116 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:59.508117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:59.508117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:59.508276 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:59.508492 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:59.508531 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 00:23:59.508614 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260924 00:23:59.509491 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:23:59.512540 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:59.509933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37086 (local address 127.0.98.62:37889)
0924 00:23:59.509997 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:59.509999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:59.510031 (+ 32us) server_negotiation.cc:407] Connection header received
0924 00:23:59.510104 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:59.510105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:59.510133 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:59.510188 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:23:59.510291 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.510736 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.511574 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.511828 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.511890 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:59.512301 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:23:59.512346 (+ 45us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 00:23:59.512489 (+ 143us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37086: token expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 00:23:59.512584 15649 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37086: token expired
W20260924 00:23:59.512717 15114 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:37889)
I20260924 00:23:59.519475 15649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:59.513035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37098 (local address 127.0.98.62:37889)
0924 00:23:59.513134 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:59.513135 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:59.513184 (+ 49us) server_negotiation.cc:407] Connection header received
0924 00:23:59.513284 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:59.513285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:59.513295 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:59.513344 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:23:59.513865 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.514395 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.515167 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.515360 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.515450 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:59.516756 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:23:59.516757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:23:59.516758 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:23:59.516765 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:23:59.518219 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:59.518652 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:59.518653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:59.518653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:59.518718 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:23:59.518931 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:23:59.518931 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:23:59.518932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:23:59.519052 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:23:59.519162 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:59.519288 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 00:23:59.519374 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 00:23:59.520774 15187 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:23:59.522656 15187 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:23:59.524986 15292 tablet_service.cc:1467] Processing CreateTablet for tablet d482bc1baae541b2999206183b70db5b (DEFAULT_TABLE table=test-table [id=97c86136cf8349fea0d08a33f8326afe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:23:59.525200 15292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d482bc1baae541b2999206183b70db5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:23:59.526711 15267 tablet_bootstrap.cc:492] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3: Bootstrap starting.
I20260924 00:23:59.527195 15267 tablet_bootstrap.cc:654] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3: Neither blocks nor log segments found. Creating new log.
I20260924 00:23:59.527976 15267 tablet_bootstrap.cc:492] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3: No bootstrap required, opened a new log
I20260924 00:23:59.528028 15267 ts_tablet_manager.cc:1397] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 00:23:59.528395 15267 raft_consensus.cc:348] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:23:59.528492 15267 raft_consensus.cc:374] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:23:59.528549 15267 raft_consensus.cc:729] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41ea6ed17b0c43af96c68ea8b238a5a3, State: Initialized, Role: FOLLOWER
I20260924 00:23:59.528606 15267 consensus_queue.cc:260] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:23:59.528697 15267 ts_tablet_manager.cc:1428] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 00:23:59.528891 15382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:59.524565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38569 (local address 127.0.0.1:50666)
0924 00:23:59.524621 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:59.524632 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:59.524670 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:59.525234 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:59.525236 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:59.525237 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:23:59.525315 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:59.525317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.526617 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.526618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:59.527568 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:59.527571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.528607 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.528608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:59.528683 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:59.528707 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 00:23:59.528743 (+ 36us) client_negotiation.cc:240] Negotiation successful
0924 00:23:59.528798 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 00:23:59.529155 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:59.524565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37727 (local address 127.0.0.1:42984)
0924 00:23:59.524902 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:59.524913 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:59.524972 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:59.525244 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:59.525245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:59.525247 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:23:59.525289 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:59.525290 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.526640 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.526641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:59.527927 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:59.527929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.528912 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.528913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:59.528992 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:59.528996 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:23:59.529050 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 00:23:59.529087 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":111,"thread_start_us":79,"threads_started":1}
I20260924 00:23:59.529309 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:59.524707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50666 (local address 127.0.98.3:38569)
0924 00:23:59.524903 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:59.524909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:59.524919 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:23:59.524960 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:59.524962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:59.524978 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:59.525096 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:23:59.525402 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.526518 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.527663 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.528502 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.529070 (+ 568us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:59.529138 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:59.529166 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 00:23:59.529212 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":98,"threads_started":1}
I20260924 00:23:59.529429 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:59.524711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.0.98.2:37727)
0924 00:23:59.524985 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:59.524990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:59.525000 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:23:59.525041 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:59.525043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:59.525065 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:59.525158 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:23:59.525372 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.526518 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.528015 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.528758 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.528818 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:59.528878 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:59.529299 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 00:23:59.529346 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":145,"threads_started":1}
I20260924 00:23:59.530026 15579 tablet_service.cc:1467] Processing CreateTablet for tablet d482bc1baae541b2999206183b70db5b (DEFAULT_TABLE table=test-table [id=97c86136cf8349fea0d08a33f8326afe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:23:59.530026 15447 tablet_service.cc:1467] Processing CreateTablet for tablet d482bc1baae541b2999206183b70db5b (DEFAULT_TABLE table=test-table [id=97c86136cf8349fea0d08a33f8326afe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:23:59.530367 15447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d482bc1baae541b2999206183b70db5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:23:59.530367 15579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d482bc1baae541b2999206183b70db5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:23:59.534027 15655 tablet_bootstrap.cc:492] T d482bc1baae541b2999206183b70db5b P dec9aec545ff4260bd04ee91f1d4d581: Bootstrap starting.
I20260924 00:23:59.534198 15656 tablet_bootstrap.cc:492] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7: Bootstrap starting.
I20260924 00:23:59.534859 15656 tablet_bootstrap.cc:654] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7: Neither blocks nor log segments found. Creating new log.
I20260924 00:23:59.534795 15655 tablet_bootstrap.cc:654] T d482bc1baae541b2999206183b70db5b P dec9aec545ff4260bd04ee91f1d4d581: Neither blocks nor log segments found. Creating new log.
I20260924 00:23:59.535523 15655 log.cc:824] T d482bc1baae541b2999206183b70db5b P dec9aec545ff4260bd04ee91f1d4d581: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:59.535523 15656 log.cc:824] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7: Log is configured to *not* fsync() on all Append() calls
I20260924 00:23:59.536397 15656 tablet_bootstrap.cc:492] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7: No bootstrap required, opened a new log
I20260924 00:23:59.536475 15656 ts_tablet_manager.cc:1397] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:23:59.536535 15655 tablet_bootstrap.cc:492] T d482bc1baae541b2999206183b70db5b P dec9aec545ff4260bd04ee91f1d4d581: No bootstrap required, opened a new log
I20260924 00:23:59.536640 15655 ts_tablet_manager.cc:1397] T d482bc1baae541b2999206183b70db5b P dec9aec545ff4260bd04ee91f1d4d581: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:23:59.538549 15656 raft_consensus.cc:348] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:23:59.538632 15655 raft_consensus.cc:348] T d482bc1baae541b2999206183b70db5b P dec9aec545ff4260bd04ee91f1d4d581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:23:59.538769 15655 raft_consensus.cc:374] T d482bc1baae541b2999206183b70db5b P dec9aec545ff4260bd04ee91f1d4d581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:23:59.538766 15656 raft_consensus.cc:374] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:23:59.538805 15656 raft_consensus.cc:729] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47ab9f4ab5794c09813b8c69f55b49b7, State: Initialized, Role: FOLLOWER
I20260924 00:23:59.538805 15655 raft_consensus.cc:729] T d482bc1baae541b2999206183b70db5b P dec9aec545ff4260bd04ee91f1d4d581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec9aec545ff4260bd04ee91f1d4d581, State: Initialized, Role: FOLLOWER
I20260924 00:23:59.538924 15655 consensus_queue.cc:260] T d482bc1baae541b2999206183b70db5b P dec9aec545ff4260bd04ee91f1d4d581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:23:59.538924 15656 consensus_queue.cc:260] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:23:59.539218 15656 ts_tablet_manager.cc:1428] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:23:59.539218 15655 ts_tablet_manager.cc:1428] T d482bc1baae541b2999206183b70db5b P dec9aec545ff4260bd04ee91f1d4d581: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:23:59.539249 15513 heartbeater.cc:503] Master 127.0.98.62:37889 was elected leader, sending a full tablet report...
I20260924 00:23:59.539335 15645 heartbeater.cc:503] Master 127.0.98.62:37889 was elected leader, sending a full tablet report...
W20260924 00:23:59.609629 15514 tablet.cc:2369] T d482bc1baae541b2999206183b70db5b P dec9aec545ff4260bd04ee91f1d4d581: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:23:59.675801 15659 raft_consensus.cc:482] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:23:59.675930 15659 raft_consensus.cc:504] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:23:59.676373 15659 leader_election.cc:288] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dec9aec545ff4260bd04ee91f1d4d581 (127.0.98.2:37727), 41ea6ed17b0c43af96c68ea8b238a5a3 (127.0.98.1:45893)
I20260924 00:23:59.679852 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:59.676683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46785 (local address 127.0.98.2:37727)
0924 00:23:59.676747 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:59.676750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:59.676762 (+ 12us) server_negotiation.cc:407] Connection header received
0924 00:23:59.676799 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:59.676801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:59.676813 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:59.676916 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:23:59.677143 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.677713 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.678960 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.679508 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.679563 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:59.679572 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:59.679769 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 00:23:59.679796 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 00:23:59.679879 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:59.676596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37727 (local address 127.0.98.3:46785)
0924 00:23:59.676659 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:59.676672 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:59.676718 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:59.676966 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:59.676967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:59.676969 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:23:59.677060 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:59.677063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.677817 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.677819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:59.678860 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:59.678863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.679595 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.679596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:59.679690 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:59.679694 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:23:59.679760 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 00:23:59.679801 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 00:23:59.680311 15467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d482bc1baae541b2999206183b70db5b" candidate_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec9aec545ff4260bd04ee91f1d4d581" is_pre_election: true
I20260924 00:23:59.680405 15383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:59.676832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58393 (local address 127.0.98.1:45893)
0924 00:23:59.676879 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:59.676880 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:59.677066 (+ 186us) server_negotiation.cc:407] Connection header received
0924 00:23:59.677094 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:59.677095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:59.677123 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:59.677185 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:23:59.677432 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.678153 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.679329 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.679949 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.680038 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:59.680138 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:59.680287 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 00:23:59.680340 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 00:23:59.680395 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:59.676737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45893 (local address 127.0.98.3:58393)
0924 00:23:59.676958 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:59.676968 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:23:59.677027 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:23:59.677202 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:23:59.677203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:23:59.677205 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:23:59.677271 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:59.677272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.678273 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.678274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:59.679235 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:23:59.679237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.680093 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.680094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:23:59.680202 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:59.680206 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:23:59.680270 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 00:23:59.680298 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":104,"threads_started":1}
I20260924 00:23:59.680510 15467 raft_consensus.cc:2454] T d482bc1baae541b2999206183b70db5b P dec9aec545ff4260bd04ee91f1d4d581 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47ab9f4ab5794c09813b8c69f55b49b7 in term 0.
I20260924 00:23:59.680792 15535 leader_election.cc:302] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47ab9f4ab5794c09813b8c69f55b49b7, dec9aec545ff4260bd04ee91f1d4d581; no voters:
I20260924 00:23:59.680856 15329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d482bc1baae541b2999206183b70db5b" candidate_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" is_pre_election: true
I20260924 00:23:59.680954 15659 raft_consensus.cc:2781] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:23:59.681016 15659 raft_consensus.cc:482] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:23:59.681051 15659 raft_consensus.cc:3037] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:23:59.681041 15329 raft_consensus.cc:2454] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47ab9f4ab5794c09813b8c69f55b49b7 in term 0.
I20260924 00:23:59.681835 15659 raft_consensus.cc:504] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:23:59.681951 15659 leader_election.cc:288] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7 [CANDIDATE]: Term 1 election: Requested vote from peers dec9aec545ff4260bd04ee91f1d4d581 (127.0.98.2:37727), 41ea6ed17b0c43af96c68ea8b238a5a3 (127.0.98.1:45893)
I20260924 00:23:59.682197 15329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d482bc1baae541b2999206183b70db5b" candidate_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3"
I20260924 00:23:59.682181 15467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d482bc1baae541b2999206183b70db5b" candidate_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dec9aec545ff4260bd04ee91f1d4d581"
I20260924 00:23:59.682274 15329 raft_consensus.cc:3037] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:23:59.682304 15467 raft_consensus.cc:3037] T d482bc1baae541b2999206183b70db5b P dec9aec545ff4260bd04ee91f1d4d581 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:23:59.683071 15329 raft_consensus.cc:2454] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47ab9f4ab5794c09813b8c69f55b49b7 in term 1.
I20260924 00:23:59.683080 15467 raft_consensus.cc:2454] T d482bc1baae541b2999206183b70db5b P dec9aec545ff4260bd04ee91f1d4d581 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47ab9f4ab5794c09813b8c69f55b49b7 in term 1.
I20260924 00:23:59.683243 15535 leader_election.cc:302] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47ab9f4ab5794c09813b8c69f55b49b7, dec9aec545ff4260bd04ee91f1d4d581; no voters:
I20260924 00:23:59.683399 15659 raft_consensus.cc:2781] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:23:59.683550 15659 raft_consensus.cc:686] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7 [term 1 LEADER]: Becoming Leader. State: Replica: 47ab9f4ab5794c09813b8c69f55b49b7, State: Running, Role: LEADER
I20260924 00:23:59.683660 15659 consensus_queue.cc:237] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:23:59.684587 15186 catalog_manager.cc:5495] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47ab9f4ab5794c09813b8c69f55b49b7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } health_report { overall_health: UNKNOWN } } }
W20260924 00:23:59.703001 15376 tablet.cc:2369] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:23:59.712765 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:59.710151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50676 (local address 127.0.98.3:38569)
0924 00:23:59.710216 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 00:23:59.710218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:23:59.710235 (+ 17us) server_negotiation.cc:407] Connection header received
0924 00:23:59.710303 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:23:59.710304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:23:59.710324 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:23:59.710386 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:23:59.710498 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.710962 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.711749 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:23:59.712008 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:23:59.712067 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:23:59.712327 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:23:59.712471 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:23:59.712578 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 00:23:59.712604 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 00:23:59.712628 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 00:23:59.714818 15329 raft_consensus.cc:1261] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Refusing update from remote peer 47ab9f4ab5794c09813b8c69f55b49b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:23:59.714818 15467 raft_consensus.cc:1261] T d482bc1baae541b2999206183b70db5b P dec9aec545ff4260bd04ee91f1d4d581 [term 1 FOLLOWER]: Refusing update from remote peer 47ab9f4ab5794c09813b8c69f55b49b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:23:59.715204 15659 consensus_queue.cc:1035] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:23:59.715368 15662 consensus_queue.cc:1035] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:23:59.717710 15665 mvcc.cc:205] Tried to move back new op lower bound from 7332697865068945408 to 7332697864944934912. Current Snapshot: MvccSnapshot[applied={T|T < 7332697865068945408}]
I20260924 00:23:59.718186 15666 mvcc.cc:205] Tried to move back new op lower bound from 7332697865068945408 to 7332697864944934912. Current Snapshot: MvccSnapshot[applied={T|T < 7332697865068945408}]
I20260924 00:23:59.720018 15187 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:23:59.720113 15187 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:23:59.720938 15187 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185: Sending DeleteTablet for 3 replicas of tablet d482bc1baae541b2999206183b70db5b
I20260924 00:23:59.721266 15447 tablet_service.cc:1514] Processing DeleteTablet for tablet d482bc1baae541b2999206183b70db5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:23:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42984
I20260924 00:23:59.721266 15579 tablet_service.cc:1514] Processing DeleteTablet for tablet d482bc1baae541b2999206183b70db5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:23:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50666
I20260924 00:23:59.721257 15292 tablet_service.cc:1514] Processing DeleteTablet for tablet d482bc1baae541b2999206183b70db5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:23:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35022
I20260924 00:23:59.721378 15384 tablet_replica.cc:330] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3: stopping tablet replica
I20260924 00:23:59.721431 15384 raft_consensus.cc:2229] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:23:59.721477 15384 raft_consensus.cc:2258] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:23:59.721563 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 15225
I20260924 00:23:59.721735 15675 tablet_replica.cc:330] T d482bc1baae541b2999206183b70db5b P dec9aec545ff4260bd04ee91f1d4d581: stopping tablet replica
I20260924 00:23:59.721828 15675 raft_consensus.cc:2229] T d482bc1baae541b2999206183b70db5b P dec9aec545ff4260bd04ee91f1d4d581 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:23:59.721925 15675 raft_consensus.cc:2258] T d482bc1baae541b2999206183b70db5b P dec9aec545ff4260bd04ee91f1d4d581 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:23:59.723518 15675 ts_tablet_manager.cc:1905] T d482bc1baae541b2999206183b70db5b P dec9aec545ff4260bd04ee91f1d4d581: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:23:59.725883 15675 ts_tablet_manager.cc:1918] T d482bc1baae541b2999206183b70db5b P dec9aec545ff4260bd04ee91f1d4d581: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:23:59.725984 15675 log.cc:1196] T d482bc1baae541b2999206183b70db5b P dec9aec545ff4260bd04ee91f1d4d581: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/wal/wals/d482bc1baae541b2999206183b70db5b
I20260924 00:23:59.726301 15675 ts_tablet_manager.cc:1939] T d482bc1baae541b2999206183b70db5b P dec9aec545ff4260bd04ee91f1d4d581: Deleting consensus metadata
I20260924 00:23:59.726773 15174 catalog_manager.cc:4841] TS dec9aec545ff4260bd04ee91f1d4d581 (127.0.98.2:37727): tablet d482bc1baae541b2999206183b70db5b (table test-table [id=97c86136cf8349fea0d08a33f8326afe]) successfully deleted
I20260924 00:23:59.728360 15676 tablet_replica.cc:330] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7: stopping tablet replica
I20260924 00:23:59.728539 15676 raft_consensus.cc:2229] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:23:59.728741 15676 raft_consensus.cc:2258] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:23:59.729276 15676 ts_tablet_manager.cc:1905] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 00:23:59.729430 15174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45893 (error 108)
I20260924 00:23:59.729816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 15381
I20260924 00:23:59.730032 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:23:59.729917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45893 (local address 127.0.0.1:42834)
0924 00:23:59.729968 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 00:23:59.729999 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 00:23:59.730397 15174 catalog_manager.cc:4584] TS 41ea6ed17b0c43af96c68ea8b238a5a3 (127.0.98.1:45893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d482bc1baae541b2999206183b70db5b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45893: connect: Connection refused (error 111)
I20260924 00:23:59.733044 15676 ts_tablet_manager.cc:1918] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:23:59.733141 15676 log.cc:1196] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/d482bc1baae541b2999206183b70db5b
I20260924 00:23:59.733456 15676 ts_tablet_manager.cc:1939] T d482bc1baae541b2999206183b70db5b P 47ab9f4ab5794c09813b8c69f55b49b7: Deleting consensus metadata
I20260924 00:23:59.733929 15172 catalog_manager.cc:4841] TS 47ab9f4ab5794c09813b8c69f55b49b7 (127.0.98.3:38569): tablet d482bc1baae541b2999206183b70db5b (table test-table [id=97c86136cf8349fea0d08a33f8326afe]) successfully deleted
I20260924 00:23:59.736634 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 15516
I20260924 00:23:59.741732 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 15154
I20260924 00:24:19.747730 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37889
--webserver_interface=127.0.98.62
--webserver_port=39923
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:19.831310 15678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:19.831532 15678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:19.831554 15678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:19.832734 15678 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 00:24:19.832772 15678 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 00:24:19.832784 15678 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 00:24:19.832796 15678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:24:19.832809 15678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:19.832823 15678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:19.832835 15678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:24:19.832847 15678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:24:19.834299 15678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:19.834507 15678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:19.834681 15678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:19.839923 15683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:19.839933 15686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:19.839946 15684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:19.839950 15678 server_base.cc:1034] running on GCE node
I20260924 00:24:19.840431 15678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:19.840739 15678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:19.841944 15678 hybrid_clock.cc:648] HybridClock initialized: now 1790209459841916 us; error 43 us; skew 500 ppm
Sep 24 00:24:19 dist-test-slave-777x krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:19.844790 15678 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:24:19.845234 15678 webserver.cc:460] Webserver started at http://127.0.98.62:39923/ using document root <none> and password file <none>
I20260924 00:24:19.845461 15678 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:19.845510 15678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:19.847039 15678 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:19.847851 15693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:19.848156 15678 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:19.848400 15678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "f2aebaa49ce74c249f086f4d9e469185"
format_stamp: "Formatted at 2026-09-24 00:23:38 on dist-test-slave-777x"
server_key: "ecb3874a00faf02d94c6b15f69a48452"
server_key_iv: "de57ad7cc71d808701a4d47ac514f40e"
server_key_version: "encryptionkey@0"
I20260924 00:24:19.848869 15678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:19.873658 15678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:19.876466 15678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:19.876636 15678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:19.881098 15678 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37889
I20260924 00:24:19.881136 15745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37889 every 8 connection(s)
I20260924 00:24:19.881564 15678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:24:19.883232 15746 sys_catalog.cc:263] Verifying existing consensus state
I20260924 00:24:19.883589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 15678
I20260924 00:24:19.883934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45893
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37889
--webserver_port=37849
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:24:19.884142 15746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185: Bootstrap starting.
I20260924 00:24:19.893327 15746 log.cc:824] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:19.895818 15746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:24:19.896210 15746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185: Bootstrap complete.
I20260924 00:24:19.898658 15746 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aebaa49ce74c249f086f4d9e469185" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } }
I20260924 00:24:19.898944 15746 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2aebaa49ce74c249f086f4d9e469185, State: Initialized, Role: FOLLOWER
I20260924 00:24:19.899171 15746 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aebaa49ce74c249f086f4d9e469185" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } }
I20260924 00:24:19.899286 15746 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:24:19.899376 15746 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:24:19.899448 15746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 2 FOLLOWER]: Advancing to term 3
I20260924 00:24:19.900851 15746 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aebaa49ce74c249f086f4d9e469185" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } }
I20260924 00:24:19.900947 15746 leader_election.cc:302] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2aebaa49ce74c249f086f4d9e469185; no voters:
I20260924 00:24:19.901218 15746 leader_election.cc:288] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 00:24:19.901255 15750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 00:24:19.901386 15750 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [term 3 LEADER]: Becoming Leader. State: Replica: f2aebaa49ce74c249f086f4d9e469185, State: Running, Role: LEADER
I20260924 00:24:19.901531 15746 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:24:19.901549 15750 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aebaa49ce74c249f086f4d9e469185" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } }
I20260924 00:24:19.901983 15751 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f2aebaa49ce74c249f086f4d9e469185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aebaa49ce74c249f086f4d9e469185" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } }
I20260924 00:24:19.902127 15751 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:19.902549 15752 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2aebaa49ce74c249f086f4d9e469185. Latest consensus state: current_term: 3 leader_uuid: "f2aebaa49ce74c249f086f4d9e469185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2aebaa49ce74c249f086f4d9e469185" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37889 } } }
I20260924 00:24:19.902628 15752 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185 [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:19.902782 15757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:24:19.903759 15757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:24:19.903934 15757 catalog_manager.cc:1259] Loaded cluster ID: b3b48ff6ba7d493f985b08ae3c92029b
I20260924 00:24:19.904132 15757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:24:19.905447 15757 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:24:19.905630 15757 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185: Loaded TSK: 0
I20260924 00:24:19.905673 15757 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f2aebaa49ce74c249f086f4d9e469185: Loaded TSK: 1
F20260924 00:24:19.905929 15757 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa65ad4c9ad google::LogMessage::Fail() at ??:0
@ 0x7fa65ad4f88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa65ad4c4de google::LogMessage::Flush() at ??:0
@ 0x7fa65ad4da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa65d4b9104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa65d4b6d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa662caa1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa662caa48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa662caa594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fa662c70c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fa662c7aad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa65cc1a0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa65cc0f954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa65da736db start_thread at ??:0
@ 0x7fa65984a61f clone at ??:0
W20260924 00:24:19.993311 15748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:19.993530 15748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:19.993554 15748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:19.994729 15748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:19.994776 15748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:19.994802 15748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:24:19.996394 15748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37849
--tserver_master_addrs=127.0.98.62:37889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:19.996636 15748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:19.996891 15748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:20.003468 15787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:20.003508 15788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:20.003712 15748 server_base.cc:1034] running on GCE node
W20260924 00:24:20.003468 15790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:20.004030 15748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:20.004366 15748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:20.005600 15748 hybrid_clock.cc:648] HybridClock initialized: now 1790209460005582 us; error 28 us; skew 500 ppm
Sep 24 00:24:20 dist-test-slave-777x krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:20.008607 15748 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:24:20.009236 15748 webserver.cc:460] Webserver started at http://127.0.98.1:37849/ using document root <none> and password file <none>
I20260924 00:24:20.009498 15748 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:20.009574 15748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:20.011199 15748 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 00:24:20.012104 15797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:20.012336 15748 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:20.012451 15748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "41ea6ed17b0c43af96c68ea8b238a5a3"
format_stamp: "Formatted at 2026-09-24 00:23:38 on dist-test-slave-777x"
server_key: "9a8971e8dd3892e4f49c7216b7cdec68"
server_key_iv: "3f00f90e86836ef438a4f6bc47c7e13e"
server_key_version: "encryptionkey@0"
I20260924 00:24:20.012956 15748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:20.035051 15748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:20.038167 15748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:20.038393 15748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:20.038720 15748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:24:20.039583 15804 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 00:24:20.041410 15748 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 00:24:20.041484 15748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 00:24:20.041531 15748 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 00:24:20.042536 15748 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 00:24:20.042589 15748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 00:24:20.042690 15804 tablet_bootstrap.cc:492] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3: Bootstrap starting.
I20260924 00:24:20.051728 15748 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45893
I20260924 00:24:20.051811 15911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45893 every 8 connection(s)
I20260924 00:24:20.052313 15748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:24:20.054646 15804 log.cc:824] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:20.056982 15804 tablet_bootstrap.cc:492] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 00:24:20.057289 15804 tablet_bootstrap.cc:492] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3: Bootstrap complete.
I20260924 00:24:20.058173 15804 ts_tablet_manager.cc:1397] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.005s
I20260924 00:24:20.059208 15804 raft_consensus.cc:348] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:24:20.059468 15804 raft_consensus.cc:729] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41ea6ed17b0c43af96c68ea8b238a5a3, State: Initialized, Role: FOLLOWER
I20260924 00:24:20.059652 15804 consensus_queue.cc:260] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:24:20.059953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 15748
I20260924 00:24:20.059967 15804 ts_tablet_manager.cc:1428] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20260924 00:24:20.060297 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37727
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37889
--webserver_port=38219
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:20.139606 15914 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:24:20.053126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.1:35697)
0924 00:24:20.053426 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:20.053518 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:20.054461 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:20.139370 (+ 84909us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":118,"threads_started":1}
W20260924 00:24:20.139976 15912 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37889 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37889: Client connection negotiation failed: client connection to 127.0.98.62:37889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 00:24:20.140489 15914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:20.140302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.1:46571)
0924 00:24:20.140361 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:20.140448 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 00:24:20.140966 15914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:20.140834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.1:33453)
0924 00:24:20.140908 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:20.140943 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 00:24:20.141139 15912 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 00:24:20.144428 15917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:20.144649 15917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:20.144701 15917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:20.145989 15917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:20.146055 15917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:20.146101 15917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:24:20.147745 15917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38219
--tserver_master_addrs=127.0.98.62:37889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:20.148029 15917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:20.148337 15917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:20.153852 15922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:20.153836 15923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:20.153851 15925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:20.154112 15917 server_base.cc:1034] running on GCE node
I20260924 00:24:20.154347 15917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:20.154620 15917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:20.155764 15917 hybrid_clock.cc:648] HybridClock initialized: now 1790209460155729 us; error 63 us; skew 500 ppm
Sep 24 00:24:20 dist-test-slave-777x krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:20.158432 15917 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:24:20.158866 15917 webserver.cc:460] Webserver started at http://127.0.98.2:38219/ using document root <none> and password file <none>
I20260924 00:24:20.159135 15917 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:20.159207 15917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:20.160799 15917 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 00:24:20.161547 15932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:20.161790 15917 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:20.161901 15917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "dec9aec545ff4260bd04ee91f1d4d581"
format_stamp: "Formatted at 2026-09-24 00:23:38 on dist-test-slave-777x"
server_key: "b88ead77189a15d78157e0a262c98d9b"
server_key_iv: "76101bc8239fbb284903c4c5e4c11c41"
server_key_version: "encryptionkey@0"
I20260924 00:24:20.162295 15917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:20.186009 15917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:20.188926 15917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:20.189105 15917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:20.189368 15917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:24:20.189692 15917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:24:20.189746 15917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:20.189779 15917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:24:20.189812 15917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:20.195752 15917 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37727
I20260924 00:24:20.195869 16045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37727 every 8 connection(s)
I20260924 00:24:20.196224 15917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:24:20.197431 16048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:20.196944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.2:54359)
0924 00:24:20.197125 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:20.197261 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":66,"threads_started":1}
W20260924 00:24:20.197897 16046 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37889 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37889: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
I20260924 00:24:20.198199 16048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:20.198097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.2:37799)
0924 00:24:20.198143 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:20.198173 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 00:24:20.198493 16048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:20.198397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.2:39817)
0924 00:24:20.198435 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:20.198476 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 00:24:20.198614 16046 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 00:24:20.206357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 15917
I20260924 00:24:20.206687 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38569
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37889
--webserver_port=45895
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:20.288093 16049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:20.288334 16049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:20.288358 16049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:20.289590 16049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:20.289635 16049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:20.289664 16049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:24:20.291249 16049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38569
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45895
--tserver_master_addrs=127.0.98.62:37889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:20.291487 16049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:20.291730 16049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:20.297230 16055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:20.297251 16054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:20.297240 16057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:20.297531 16049 server_base.cc:1034] running on GCE node
I20260924 00:24:20.297762 16049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:20.298022 16049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:20.299206 16049 hybrid_clock.cc:648] HybridClock initialized: now 1790209460299192 us; error 44 us; skew 500 ppm
Sep 24 00:24:20 dist-test-slave-777x krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:20.301981 16049 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:24:20.302395 16049 webserver.cc:460] Webserver started at http://127.0.98.3:45895/ using document root <none> and password file <none>
I20260924 00:24:20.302575 16049 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:20.302616 16049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:20.303982 16049 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:20.304782 16064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:20.304981 16049 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:24:20.305068 16049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "47ab9f4ab5794c09813b8c69f55b49b7"
format_stamp: "Formatted at 2026-09-24 00:23:38 on dist-test-slave-777x"
server_key: "11e113b9a001608a43d35e119f4dbbb8"
server_key_iv: "15380b16c0a289ce45150abaa8a97c98"
server_key_version: "encryptionkey@0"
I20260924 00:24:20.305410 16049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:20.329015 16049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:20.331663 16049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:20.331835 16049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:20.332067 16049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:24:20.332494 16049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:24:20.332533 16049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:20.332559 16049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:24:20.332613 16049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:20.339042 16049 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38569
I20260924 00:24:20.339129 16177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38569 every 8 connection(s)
I20260924 00:24:20.339463 16049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:24:20.340693 16180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:20.340315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.3:39747)
0924 00:24:20.340482 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:20.340558 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
W20260924 00:24:20.341161 16178 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37889 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37889: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
I20260924 00:24:20.341470 16180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:20.341373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.3:35215)
0924 00:24:20.341421 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:20.341452 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 00:24:20.341784 16180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:20.341698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.3:57547)
0924 00:24:20.341738 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:20.341761 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 00:24:20.341943 16178 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 00:24:20.342346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 16049
I20260924 00:24:21.141831 16182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:21.141538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.1:46583)
0924 00:24:21.141713 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:21.141761 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":78,"threads_started":1}
I20260924 00:24:21.199318 16183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:21.199034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.2:49329)
0924 00:24:21.199190 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:21.199242 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
I20260924 00:24:21.342531 16184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:21.342308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.3:39415)
0924 00:24:21.342432 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:21.342472 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260924 00:24:21.575922 16185 raft_consensus.cc:482] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:24:21.576102 16185 raft_consensus.cc:504] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:24:21.576576 16185 leader_election.cc:288] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dec9aec545ff4260bd04ee91f1d4d581 (127.0.98.2:37727), 47ab9f4ab5794c09813b8c69f55b49b7 (127.0.98.3:38569)
Sep 24 00:24:21 dist-test-slave-777x krb5kdc[14605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 24 00:24:21 dist-test-slave-777x krb5kdc[14605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260924 00:24:21.588203 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:21.576999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34183 (local address 127.0.98.3:38569)
0924 00:24:21.577145 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:21.577155 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:21.577195 (+ 40us) server_negotiation.cc:407] Connection header received
0924 00:24:21.577667 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:21.577670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:21.577726 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:21.577896 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:21.578510 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:21.579421 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:21.580584 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:21.581026 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:21.581162 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:21.584278 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:21.584283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:21.584284 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:21.584299 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:21.586198 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:21.586678 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:21.586684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:21.586684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:21.586787 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:21.587087 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:21.587088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:21.587088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:21.587482 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:21.587555 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:21.587917 (+ 362us) server_negotiation.cc:299] Negotiation successful
0924 00:24:21.588058 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":86,"threads_started":1}
I20260924 00:24:21.588191 16182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:21.576718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37727 (local address 127.0.98.1:36213)
0924 00:24:21.576759 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:21.576772 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:21.576852 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:21.577766 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:21.577769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:21.578173 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:24:21.578376 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:21.578382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:21.579505 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:21.579506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:21.580456 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:21.580458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:21.581132 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:21.581133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:21.581264 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:21.581982 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:24:21.581986 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:24:21.583713 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:24:21.586159 (+ 2446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:24:21.586167 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:24:21.586170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:24:21.586567 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:24:21.586853 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:24:21.586854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:24:21.586854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:24:21.586979 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:24:21.587609 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:24:21.587610 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:24:21.587708 (+ 98us) client_negotiation.cc:769] Sending connection context
0924 00:24:21.587807 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 00:24:21.588049 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":15}
I20260924 00:24:21.588222 16187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:21.576866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36213 (local address 127.0.98.2:37727)
0924 00:24:21.577018 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:21.577030 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:21.577072 (+ 42us) server_negotiation.cc:407] Connection header received
0924 00:24:21.577526 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:21.577528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:21.577572 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:21.577688 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:21.578513 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:21.579384 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:21.580560 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:21.581026 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:21.581162 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:21.583897 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:21.583902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:21.583903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:21.583917 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:21.586050 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:21.586666 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:21.586670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:21.586670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:21.586776 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:21.587139 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:21.587140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:21.587140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:21.587500 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:21.587619 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:21.587891 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 00:24:21.588042 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260924 00:24:21.588378 16186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:21.576740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38569 (local address 127.0.98.1:34183)
0924 00:24:21.576870 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:21.576880 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:21.576935 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:21.577888 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:21.577889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:21.578173 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:24:21.578376 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:21.578388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:21.579575 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:21.579576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:21.580457 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:21.580460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:21.581132 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:21.581133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:21.581272 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:21.581914 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:24:21.581923 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:24:21.584080 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:24:21.586359 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:24:21.586360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:24:21.586360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:24:21.586568 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:24:21.586884 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:24:21.586885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:24:21.586885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:24:21.587002 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:24:21.587559 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:24:21.587567 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:24:21.587728 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 00:24:21.587839 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 00:24:21.588158 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":228,"thread_start_us":62,"threads_started":1}
W20260924 00:24:21.589394 15799 leader_election.cc:341] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47ab9f4ab5794c09813b8c69f55b49b7 (127.0.98.3:38569): Not found: Tablet not found: d482bc1baae541b2999206183b70db5b
W20260924 00:24:21.589499 15801 leader_election.cc:341] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dec9aec545ff4260bd04ee91f1d4d581 (127.0.98.2:37727): Not found: Tablet not found: d482bc1baae541b2999206183b70db5b
I20260924 00:24:21.589567 15801 leader_election.cc:302] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41ea6ed17b0c43af96c68ea8b238a5a3; no voters: 47ab9f4ab5794c09813b8c69f55b49b7, dec9aec545ff4260bd04ee91f1d4d581
I20260924 00:24:21.589668 16185 raft_consensus.cc:2726] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:24:22.142642 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:22.142383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.1:40095)
0924 00:24:22.142550 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:22.142595 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":72,"threads_started":1}
I20260924 00:24:22.200069 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:22.199851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.2:56477)
0924 00:24:22.199981 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:22.200026 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260924 00:24:22.343390 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:22.343057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.3:35717)
0924 00:24:22.343203 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:22.343280 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":66,"threads_started":1}
I20260924 00:24:23.143486 16192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:23.143180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.1:50351)
0924 00:24:23.143344 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:23.143408 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":68,"threads_started":1}
I20260924 00:24:23.200845 16193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:23.200617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.2:43221)
0924 00:24:23.200749 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:23.200793 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260924 00:24:23.344259 16194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:23.343985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.3:54507)
0924 00:24:23.344118 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:23.344159 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260924 00:24:23.813756 16195 raft_consensus.cc:482] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:24:23.813861 16195 raft_consensus.cc:504] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:24:23.814044 16195 leader_election.cc:288] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dec9aec545ff4260bd04ee91f1d4d581 (127.0.98.2:37727), 47ab9f4ab5794c09813b8c69f55b49b7 (127.0.98.3:38569)
W20260924 00:24:23.814544 15799 leader_election.cc:341] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47ab9f4ab5794c09813b8c69f55b49b7 (127.0.98.3:38569): Not found: Tablet not found: d482bc1baae541b2999206183b70db5b
W20260924 00:24:23.814644 15801 leader_election.cc:341] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dec9aec545ff4260bd04ee91f1d4d581 (127.0.98.2:37727): Not found: Tablet not found: d482bc1baae541b2999206183b70db5b
I20260924 00:24:23.814723 15801 leader_election.cc:302] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41ea6ed17b0c43af96c68ea8b238a5a3; no voters: 47ab9f4ab5794c09813b8c69f55b49b7, dec9aec545ff4260bd04ee91f1d4d581
I20260924 00:24:23.814810 16195 raft_consensus.cc:2726] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:24:24.144505 16196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:24.144158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.1:60147)
0924 00:24:24.144402 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:24.144453 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":143,"threads_started":1}
I20260924 00:24:24.201601 16197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:24.201321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.2:57743)
0924 00:24:24.201503 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:24.201552 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":80,"threads_started":1}
I20260924 00:24:24.345252 16198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:24.344997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.3:42125)
0924 00:24:24.345150 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:24.345194 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260924 00:24:25.145431 16199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:25.145133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.1:43515)
0924 00:24:25.145288 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:25.145324 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260924 00:24:25.202396 16200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:25.202173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.2:43261)
0924 00:24:25.202306 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:25.202350 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260924 00:24:25.346184 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:25.345935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.3:34455)
0924 00:24:25.346090 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:25.346134 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":66,"threads_started":1}
I20260924 00:24:26.146235 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:26.145999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.1:36663)
0924 00:24:26.146143 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:26.146188 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
I20260924 00:24:26.203280 16203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:26.202944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.2:50261)
0924 00:24:26.203194 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:26.203234 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":208,"threads_started":1}
I20260924 00:24:26.346985 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:26.346740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.3:45137)
0924 00:24:26.346891 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:26.346939 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":62,"threads_started":1}
I20260924 00:24:26.446617 16205 raft_consensus.cc:482] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:24:26.446827 16205 raft_consensus.cc:504] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:24:26.447043 16205 leader_election.cc:288] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dec9aec545ff4260bd04ee91f1d4d581 (127.0.98.2:37727), 47ab9f4ab5794c09813b8c69f55b49b7 (127.0.98.3:38569)
W20260924 00:24:26.447428 15799 leader_election.cc:341] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47ab9f4ab5794c09813b8c69f55b49b7 (127.0.98.3:38569): Not found: Tablet not found: d482bc1baae541b2999206183b70db5b
W20260924 00:24:26.447486 15801 leader_election.cc:341] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dec9aec545ff4260bd04ee91f1d4d581 (127.0.98.2:37727): Not found: Tablet not found: d482bc1baae541b2999206183b70db5b
I20260924 00:24:26.447520 15801 leader_election.cc:302] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41ea6ed17b0c43af96c68ea8b238a5a3; no voters: 47ab9f4ab5794c09813b8c69f55b49b7, dec9aec545ff4260bd04ee91f1d4d581
I20260924 00:24:26.447630 16205 raft_consensus.cc:2726] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:24:27.146965 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:27.146712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.1:41929)
0924 00:24:27.146845 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:27.146916 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260924 00:24:27.204089 16207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:27.203824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.2:33935)
0924 00:24:27.203999 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:27.204042 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":106,"threads_started":1}
I20260924 00:24:27.347764 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:27.347505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.3:36055)
0924 00:24:27.347652 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:27.347698 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
I20260924 00:24:28.147745 16209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:28.147495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.1:40437)
0924 00:24:28.147657 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:28.147695 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":68,"threads_started":1}
I20260924 00:24:28.204879 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:28.204651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.2:60527)
0924 00:24:28.204785 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:28.204830 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260924 00:24:28.348544 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:28.348294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.3:34131)
0924 00:24:28.348453 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:28.348492 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
I20260924 00:24:28.596401 16212 raft_consensus.cc:482] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:24:28.596525 16212 raft_consensus.cc:504] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:24:28.596737 16212 leader_election.cc:288] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dec9aec545ff4260bd04ee91f1d4d581 (127.0.98.2:37727), 47ab9f4ab5794c09813b8c69f55b49b7 (127.0.98.3:38569)
W20260924 00:24:28.597208 15799 leader_election.cc:341] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47ab9f4ab5794c09813b8c69f55b49b7 (127.0.98.3:38569): Not found: Tablet not found: d482bc1baae541b2999206183b70db5b
W20260924 00:24:28.597296 15801 leader_election.cc:341] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dec9aec545ff4260bd04ee91f1d4d581 (127.0.98.2:37727): Not found: Tablet not found: d482bc1baae541b2999206183b70db5b
I20260924 00:24:28.597340 15801 leader_election.cc:302] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41ea6ed17b0c43af96c68ea8b238a5a3; no voters: 47ab9f4ab5794c09813b8c69f55b49b7, dec9aec545ff4260bd04ee91f1d4d581
I20260924 00:24:28.597425 16212 raft_consensus.cc:2726] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:24:29.148610 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:29.148330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.1:34487)
0924 00:24:29.148506 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:29.148558 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
I20260924 00:24:29.205770 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:29.205516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.2:41493)
0924 00:24:29.205674 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:29.205721 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":70,"threads_started":1}
I20260924 00:24:29.349238 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:29.348978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.3:35889)
0924 00:24:29.349134 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:29.349176 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":68,"threads_started":1}
I20260924 00:24:30.149475 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:30.149245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.1:55987)
0924 00:24:30.149375 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:30.149424 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260924 00:24:30.206571 16217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:30.206339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.2:53697)
0924 00:24:30.206477 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:30.206525 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260924 00:24:30.350068 16218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:30.349811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.3:43207)
0924 00:24:30.349973 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:30.350017 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":63,"threads_started":1}
I20260924 00:24:31.150352 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:31.150073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.1:47579)
0924 00:24:31.150254 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:31.150302 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":85,"threads_started":1}
I20260924 00:24:31.207465 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:31.207185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.2:49367)
0924 00:24:31.207338 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:31.207384 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260924 00:24:31.350937 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:31.350666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.3:42267)
0924 00:24:31.350825 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:31.350871 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
I20260924 00:24:32.151183 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:32.150921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.1:58491)
0924 00:24:32.151078 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:32.151134 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":80,"threads_started":1}
I20260924 00:24:32.208444 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:32.208135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.2:57383)
0924 00:24:32.208339 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:32.208396 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":123,"threads_started":1}
I20260924 00:24:32.351760 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:32.351508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.3:35721)
0924 00:24:32.351665 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:32.351714 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
I20260924 00:24:33.152081 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:33.151755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.1:47071)
0924 00:24:33.151974 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:33.152030 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":123,"threads_started":1}
I20260924 00:24:33.209250 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:33.209029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.2:36219)
0924 00:24:33.209162 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:33.209205 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260924 00:24:33.352716 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:33.352381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.3:59699)
0924 00:24:33.352612 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:33.352667 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":71,"threads_started":1}
I20260924 00:24:33.710742 16228 raft_consensus.cc:482] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:24:33.710872 16228 raft_consensus.cc:504] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dec9aec545ff4260bd04ee91f1d4d581" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37727 } } peers { permanent_uuid: "47ab9f4ab5794c09813b8c69f55b49b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38569 } } peers { permanent_uuid: "41ea6ed17b0c43af96c68ea8b238a5a3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45893 } }
I20260924 00:24:33.711071 16228 leader_election.cc:288] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dec9aec545ff4260bd04ee91f1d4d581 (127.0.98.2:37727), 47ab9f4ab5794c09813b8c69f55b49b7 (127.0.98.3:38569)
W20260924 00:24:33.711483 15799 leader_election.cc:341] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 47ab9f4ab5794c09813b8c69f55b49b7 (127.0.98.3:38569): Not found: Tablet not found: d482bc1baae541b2999206183b70db5b
W20260924 00:24:33.711565 15801 leader_election.cc:341] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dec9aec545ff4260bd04ee91f1d4d581 (127.0.98.2:37727): Not found: Tablet not found: d482bc1baae541b2999206183b70db5b
I20260924 00:24:33.711678 15801 leader_election.cc:302] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41ea6ed17b0c43af96c68ea8b238a5a3; no voters: 47ab9f4ab5794c09813b8c69f55b49b7, dec9aec545ff4260bd04ee91f1d4d581
I20260924 00:24:33.711758 16228 raft_consensus.cc:2726] T d482bc1baae541b2999206183b70db5b P 41ea6ed17b0c43af96c68ea8b238a5a3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 00:24:34.152866 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:34.152642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.1:39187)
0924 00:24:34.152776 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:34.152819 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260924 00:24:34.210021 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:34.209783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.2:51923)
0924 00:24:34.209928 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:34.209972 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260924 00:24:34.353489 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:34.353259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.3:55931)
0924 00:24:34.353399 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:34.353442 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260924 00:24:35.153811 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:35.153562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.1:52671)
0924 00:24:35.153718 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:35.153763 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":63,"threads_started":1}
I20260924 00:24:35.211002 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:35.210730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.2:52487)
0924 00:24:35.210914 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:35.210951 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":99,"threads_started":1}
I20260924 00:24:35.346190 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 15748
I20260924 00:24:35.351365 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 15917
I20260924 00:24:35.354331 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:35.354087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37889 (local address 127.0.98.3:44149)
0924 00:24:35.354249 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:35.354286 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260924 00:24:35.356027 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 16049
2026-09-24T00:24:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58486 ms)
[----------] 30 tests from SecurityITest (321045 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:24:35 dist-test-slave-777x krb5kdc[16237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:24:35 dist-test-slave-777x krb5kdc[16237](info): set up 2 sockets
Sep 24 00:24:35 dist-test-slave-777x krb5kdc[16237](info): commencing operation
krb5kdc: starting...
W20260924 00:24:36.415951 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:24:36 dist-test-slave-777x krb5kdc[16237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:24:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:24:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:24:36.544139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37621
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:36.633953 16250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:36.634213 16250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:36.634251 16250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:36.635419 16250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:24:36.635495 16250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:36.635522 16250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:36.635550 16250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:24:36.635581 16250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:24:36.637068 16250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:36.637351 16250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:36.637584 16250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:36.642500 16255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:36.642514 16258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:36.642506 16256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:36.642741 16250 server_base.cc:1034] running on GCE node
I20260924 00:24:36.643070 16250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:36.643384 16250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:36.644526 16250 hybrid_clock.cc:648] HybridClock initialized: now 1790209476644510 us; error 26 us; skew 500 ppm
Sep 24 00:24:36 dist-test-slave-777x krb5kdc[16237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:36.647177 16250 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:24:36.647630 16250 webserver.cc:460] Webserver started at http://127.0.98.62:37449/ using document root <none> and password file <none>
I20260924 00:24:36.647866 16250 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:36.647941 16250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:36.648095 16250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:36.649305 16250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "53c9ed0ccc234ff4a0b231ff82720c3f"
format_stamp: "Formatted at 2026-09-24 00:24:36 on dist-test-slave-777x"
server_key: "262bc0037089c798f6447b989ffaf8e9"
server_key_iv: "c7e391e04caadae4b8a0659f29bef2d8"
server_key_version: "encryptionkey@0"
I20260924 00:24:36.649689 16250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "53c9ed0ccc234ff4a0b231ff82720c3f"
format_stamp: "Formatted at 2026-09-24 00:24:36 on dist-test-slave-777x"
server_key: "262bc0037089c798f6447b989ffaf8e9"
server_key_iv: "c7e391e04caadae4b8a0659f29bef2d8"
server_key_version: "encryptionkey@0"
I20260924 00:24:36.651067 16250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:24:36.651903 16265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:36.652148 16250 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:36.652300 16250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "53c9ed0ccc234ff4a0b231ff82720c3f"
format_stamp: "Formatted at 2026-09-24 00:24:36 on dist-test-slave-777x"
server_key: "262bc0037089c798f6447b989ffaf8e9"
server_key_iv: "c7e391e04caadae4b8a0659f29bef2d8"
server_key_version: "encryptionkey@0"
I20260924 00:24:36.652386 16250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:36.686623 16250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:36.689214 16250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:36.689399 16250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:36.693413 16250 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37621
I20260924 00:24:36.693457 16317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37621 every 8 connection(s)
I20260924 00:24:36.693894 16250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:24:36.694835 16318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:36.697274 16318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f: Bootstrap starting.
I20260924 00:24:36.697894 16318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:36.698186 16318 log.cc:824] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:36.698937 16318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f: No bootstrap required, opened a new log
I20260924 00:24:36.700017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 16250
I20260924 00:24:36.700145 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:24:36.700399 392 external_mini_cluster.cc:1428] Setting key 0c01ea295aa3edb2dc6e51b2b5d0d2c3
I20260924 00:24:36.700891 16318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c9ed0ccc234ff4a0b231ff82720c3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37621 } }
I20260924 00:24:36.701092 16318 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:36.701179 16318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c9ed0ccc234ff4a0b231ff82720c3f, State: Initialized, Role: FOLLOWER
I20260924 00:24:36.701350 16318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c9ed0ccc234ff4a0b231ff82720c3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37621 } }
I20260924 00:24:36.701454 16318 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:24:36.701502 16318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:24:36.701552 16318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:36.702705 16318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c9ed0ccc234ff4a0b231ff82720c3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37621 } }
I20260924 00:24:36.702797 16318 leader_election.cc:302] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53c9ed0ccc234ff4a0b231ff82720c3f; no voters:
I20260924 00:24:36.702963 16318 leader_election.cc:288] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:24:36.702986 16323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:36.703096 16323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f [term 1 LEADER]: Becoming Leader. State: Replica: 53c9ed0ccc234ff4a0b231ff82720c3f, State: Running, Role: LEADER
I20260924 00:24:36.703223 16318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:24:36.703246 16323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c9ed0ccc234ff4a0b231ff82720c3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37621 } }
I20260924 00:24:36.703641 16323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53c9ed0ccc234ff4a0b231ff82720c3f. Latest consensus state: current_term: 1 leader_uuid: "53c9ed0ccc234ff4a0b231ff82720c3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c9ed0ccc234ff4a0b231ff82720c3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37621 } } }
I20260924 00:24:36.703655 16325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53c9ed0ccc234ff4a0b231ff82720c3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c9ed0ccc234ff4a0b231ff82720c3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37621 } } }
I20260924 00:24:36.703836 16323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:36.703852 16325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:36.704080 16332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:24:36.704643 16332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:24:36.706372 16332 catalog_manager.cc:1347] Generated new cluster ID: a989f28b1641462eb8de729a729cfd26
I20260924 00:24:36.706427 16332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:24:36 dist-test-slave-777x krb5kdc[16237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:24:36.710435 16332 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:24:36.711191 16332 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:24:36.711453 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:36.701016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42940 (local address 127.0.98.62:37621)
0924 00:24:36.701215 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:36.701235 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:36.701259 (+ 24us) server_negotiation.cc:407] Connection header received
0924 00:24:36.701401 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:36.701410 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:36.701459 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:36.701614 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:36.702417 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:36.703671 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:36.704898 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:36.705504 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:36.705644 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:36.708082 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:36.708085 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:36.708086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:36.708100 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:36.709878 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:36.710310 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:36.710312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:36.710313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:36.710415 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:36.710621 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:36.710622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:36.710622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:36.710849 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:36.711012 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:36.711106 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 00:24:36.711262 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":94,"threads_started":1}
I20260924 00:24:36.723973 16332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f: Generated new TSK 0
I20260924 00:24:36.724215 16332 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:24:36.760334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37621
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:36.863188 16344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:36.863499 16344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:36.863546 16344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:36.865047 16344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:36.865118 16344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:36.865172 16344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:24:36.866874 16344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:36.867165 16344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:36.867437 16344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:36.873873 16350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:36.873946 16349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:36.874137 16352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:36.874105 16344 server_base.cc:1034] running on GCE node
I20260924 00:24:36.874370 16344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:36.874645 16344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:36.875830 16344 hybrid_clock.cc:648] HybridClock initialized: now 1790209476875806 us; error 36 us; skew 500 ppm
Sep 24 00:24:36 dist-test-slave-777x krb5kdc[16237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:36.878907 16344 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:24:36.879388 16344 webserver.cc:460] Webserver started at http://127.0.98.1:37033/ using document root <none> and password file <none>
I20260924 00:24:36.879673 16344 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:36.879741 16344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:36.879870 16344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:36.880987 16344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "f36e27af724642b0a971c16ce110b39e"
format_stamp: "Formatted at 2026-09-24 00:24:36 on dist-test-slave-777x"
server_key: "68dbbd817656fe67799f3949b10d1b5d"
server_key_iv: "704c347c2455fa399a796ce6c30ad75f"
server_key_version: "encryptionkey@0"
I20260924 00:24:36.881353 16344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f36e27af724642b0a971c16ce110b39e"
format_stamp: "Formatted at 2026-09-24 00:24:36 on dist-test-slave-777x"
server_key: "68dbbd817656fe67799f3949b10d1b5d"
server_key_iv: "704c347c2455fa399a796ce6c30ad75f"
server_key_version: "encryptionkey@0"
I20260924 00:24:36.882758 16344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 00:24:36.883508 16359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:36.883708 16344 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:24:36.883832 16344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "f36e27af724642b0a971c16ce110b39e"
format_stamp: "Formatted at 2026-09-24 00:24:36 on dist-test-slave-777x"
server_key: "68dbbd817656fe67799f3949b10d1b5d"
server_key_iv: "704c347c2455fa399a796ce6c30ad75f"
server_key_version: "encryptionkey@0"
I20260924 00:24:36.883909 16344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:36.913439 16344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:36.915939 16344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:36.916116 16344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:36.916419 16344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:24:36.916775 16344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:24:36.916844 16344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:36.916889 16344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:24:36.916916 16344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:36.922796 16344 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41677
I20260924 00:24:36.922879 16472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41677 every 8 connection(s)
I20260924 00:24:36.923225 16344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:24:36.927060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 16344
I20260924 00:24:36.927176 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:24:36.927377 392 external_mini_cluster.cc:1428] Setting key 42f197ab5c7cd44d53b513639b273177
Sep 24 00:24:36 dist-test-slave-777x krb5kdc[16237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209476, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:24:36.933971 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:36.924099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44813 (local address 127.0.98.62:37621)
0924 00:24:36.924187 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:36.924189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:36.924438 (+ 249us) server_negotiation.cc:407] Connection header received
0924 00:24:36.925443 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:36.925444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:36.925458 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:36.925522 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:36.926153 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:36.926664 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:36.928099 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:36.928384 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:36.928468 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:36.931018 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:36.931019 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:36.931020 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:36.931027 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:36.932560 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:36.932989 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:36.932990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:36.932991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:36.933051 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:36.933315 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:36.933315 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:36.933316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:36.933438 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:36.933512 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:36.933790 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 00:24:36.933879 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 00:24:36.934172 16475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:36.924081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37621 (local address 127.0.98.1:44813)
0924 00:24:36.924318 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:36.924356 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:36.925322 (+ 966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:36.925600 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:36.925604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:36.925860 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:24:36.926052 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:36.926057 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:36.926763 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:36.926764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:36.928001 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:36.928004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:36.928479 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:36.928480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:36.928605 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:36.929137 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:24:36.929146 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:24:36.930920 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:24:36.932658 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:24:36.932665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:24:36.932668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:24:36.932898 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:24:36.933125 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:24:36.933126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:24:36.933127 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:24:36.933239 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:24:36.933504 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:24:36.933510 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:24:36.933652 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 00:24:36.933771 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 00:24:36.933931 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":127,"threads_started":1}
I20260924 00:24:36.934736 16473 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37621
I20260924 00:24:36.934828 16473 heartbeater.cc:464] Registering TS with master...
I20260924 00:24:36.935024 16473 heartbeater.cc:511] Master 127.0.98.62:37621 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 00:24:36.935652 16282 ts_manager.cc:194] Registered new tserver with Master: f36e27af724642b0a971c16ce110b39e (127.0.98.1:41677)
I20260924 00:24:36.936427 16282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:24:36.953118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37621
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:37.033597 16478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:37.033802 16478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:37.033824 16478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:37.034967 16478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:37.035008 16478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:37.035034 16478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:24:37.036535 16478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:37.036774 16478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:37.037029 16478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:37.041988 16484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:37.041990 16486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:37.041975 16483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:37.042284 16478 server_base.cc:1034] running on GCE node
I20260924 00:24:37.042502 16478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:37.042747 16478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:37.043917 16478 hybrid_clock.cc:648] HybridClock initialized: now 1790209477043900 us; error 27 us; skew 500 ppm
Sep 24 00:24:37 dist-test-slave-777x krb5kdc[16237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:37.046530 16478 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:24:37.046960 16478 webserver.cc:460] Webserver started at http://127.0.98.2:38493/ using document root <none> and password file <none>
I20260924 00:24:37.047181 16478 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:37.047246 16478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:37.047410 16478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:37.048580 16478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "d8e56d382e9245ba8f563b2dc455552f"
format_stamp: "Formatted at 2026-09-24 00:24:37 on dist-test-slave-777x"
server_key: "40cb497b6b99f8804abd94ad52a92a3f"
server_key_iv: "b3a91c8e360d8f3500391b2187f0f064"
server_key_version: "encryptionkey@0"
I20260924 00:24:37.048954 16478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d8e56d382e9245ba8f563b2dc455552f"
format_stamp: "Formatted at 2026-09-24 00:24:37 on dist-test-slave-777x"
server_key: "40cb497b6b99f8804abd94ad52a92a3f"
server_key_iv: "b3a91c8e360d8f3500391b2187f0f064"
server_key_version: "encryptionkey@0"
I20260924 00:24:37.050278 16478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:24:37.051026 16493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:37.051201 16478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:24:37.051321 16478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "d8e56d382e9245ba8f563b2dc455552f"
format_stamp: "Formatted at 2026-09-24 00:24:37 on dist-test-slave-777x"
server_key: "40cb497b6b99f8804abd94ad52a92a3f"
server_key_iv: "b3a91c8e360d8f3500391b2187f0f064"
server_key_version: "encryptionkey@0"
I20260924 00:24:37.051395 16478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:37.060216 16478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:37.062481 16478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:37.062638 16478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:37.062866 16478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:24:37.063156 16478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:24:37.063208 16478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:37.063248 16478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:24:37.063313 16478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:37.069315 16478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33145
I20260924 00:24:37.069394 16606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33145 every 8 connection(s)
I20260924 00:24:37.069703 16478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 00:24:37 dist-test-slave-777x krb5kdc[16237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:24:37.077936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 16478
I20260924 00:24:37.078055 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:24:37.078262 392 external_mini_cluster.cc:1428] Setting key 6ae1635141b3d2aa6097be8778830015
I20260924 00:24:37.079708 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:37.070443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47903 (local address 127.0.98.62:37621)
0924 00:24:37.070486 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:37.070488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:37.070707 (+ 219us) server_negotiation.cc:407] Connection header received
0924 00:24:37.071671 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:37.071673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:37.071686 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:37.071740 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:37.072445 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.072940 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.073953 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.074143 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.074227 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:37.076413 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:37.076415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:37.076415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:37.076422 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:37.078024 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:37.078558 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:37.078559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:37.078560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:37.078621 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:37.078925 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:37.078926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:37.078926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:37.079084 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:37.079156 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:37.079455 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 00:24:37.079553 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:24:37.079746 16609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:37.070416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37621 (local address 127.0.98.2:47903)
0924 00:24:37.070588 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:37.070623 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:37.071553 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:37.071828 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:37.071834 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:37.072093 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:24:37.072347 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:37.072352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.073031 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.073032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:37.073848 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:37.073850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.074234 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.074235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:37.074370 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:37.074881 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:24:37.074889 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:24:37.076331 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:24:37.078153 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:24:37.078160 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:24:37.078162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:24:37.078457 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:24:37.078697 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:24:37.078698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:24:37.078698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:24:37.078837 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:24:37.079158 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:24:37.079163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:24:37.079303 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 00:24:37.079393 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 00:24:37.079515 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
I20260924 00:24:37.080212 16607 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37621
I20260924 00:24:37.080319 16607 heartbeater.cc:464] Registering TS with master...
I20260924 00:24:37.080518 16607 heartbeater.cc:511] Master 127.0.98.62:37621 requested a full tablet report, sending...
I20260924 00:24:37.080941 16282 ts_manager.cc:194] Registered new tserver with Master: d8e56d382e9245ba8f563b2dc455552f (127.0.98.2:33145)
I20260924 00:24:37.081327 16282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47903
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:24:37.103577 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37621
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:37.185012 16612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:37.185271 16612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:37.185312 16612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:37.186537 16612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:37.186609 16612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:37.186654 16612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:24:37.188319 16612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:37.188616 16612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:37.188869 16612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:37.194125 16620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:37.194125 16618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:37.194185 16617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:37.194442 16612 server_base.cc:1034] running on GCE node
I20260924 00:24:37.194716 16612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:37.195025 16612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:37.196251 16612 hybrid_clock.cc:648] HybridClock initialized: now 1790209477196215 us; error 43 us; skew 500 ppm
Sep 24 00:24:37 dist-test-slave-777x krb5kdc[16237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:37.199311 16612 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:24:37.199719 16612 webserver.cc:460] Webserver started at http://127.0.98.3:40725/ using document root <none> and password file <none>
I20260924 00:24:37.199976 16612 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:37.200026 16612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:37.200117 16612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:37.201601 16612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "62d5fa98042b467a83ba0b2594e3ba4e"
format_stamp: "Formatted at 2026-09-24 00:24:37 on dist-test-slave-777x"
server_key: "1c16e02418434f1aa2f310e9ef25203e"
server_key_iv: "77de39ad7bdcba237aa0f037ff80427c"
server_key_version: "encryptionkey@0"
I20260924 00:24:37.201988 16612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "62d5fa98042b467a83ba0b2594e3ba4e"
format_stamp: "Formatted at 2026-09-24 00:24:37 on dist-test-slave-777x"
server_key: "1c16e02418434f1aa2f310e9ef25203e"
server_key_iv: "77de39ad7bdcba237aa0f037ff80427c"
server_key_version: "encryptionkey@0"
I20260924 00:24:37.203579 16612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 00:24:37.204541 16627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:37.204751 16612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:37.204864 16612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "62d5fa98042b467a83ba0b2594e3ba4e"
format_stamp: "Formatted at 2026-09-24 00:24:37 on dist-test-slave-777x"
server_key: "1c16e02418434f1aa2f310e9ef25203e"
server_key_iv: "77de39ad7bdcba237aa0f037ff80427c"
server_key_version: "encryptionkey@0"
I20260924 00:24:37.204979 16612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:37.226653 16612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:37.228734 16612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:37.228901 16612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:37.229141 16612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:24:37.229468 16612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:24:37.229525 16612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:37.229568 16612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:24:37.229597 16612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:37.235430 16612 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42597
I20260924 00:24:37.235508 16740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42597 every 8 connection(s)
I20260924 00:24:37.235836 16612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:24:37.238415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 16612
I20260924 00:24:37.238520 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:24:37.238754 392 external_mini_cluster.cc:1428] Setting key 363cca0e3269653088d93ac3c50f0a14
Sep 24 00:24:37 dist-test-slave-777x krb5kdc[16237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209477, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:24:37.246224 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:37.236618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41803 (local address 127.0.98.62:37621)
0924 00:24:37.236688 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:37.236690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:37.236949 (+ 259us) server_negotiation.cc:407] Connection header received
0924 00:24:37.237846 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:37.237847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:37.237857 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:37.237919 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:37.238786 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.239480 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.240792 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.240982 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.241040 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:37.243448 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:37.243449 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:37.243449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:37.243456 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:37.244969 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:37.245330 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:37.245331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:37.245331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:37.245383 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:37.245603 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:37.245604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:37.245604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:37.245718 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:37.245772 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:37.246073 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 00:24:37.246154 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 00:24:37.246296 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:37.236592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37621 (local address 127.0.98.3:41803)
0924 00:24:37.236770 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:37.236829 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:37.237731 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:37.237994 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:37.237998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:37.238364 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:24:37.238666 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:37.238673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.239590 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.239592 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:37.240686 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:37.240689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.241057 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.241058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:37.241163 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:37.241785 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:24:37.241794 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:24:37.243341 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:24:37.245038 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:24:37.245044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:24:37.245046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:24:37.245255 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:24:37.245445 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:24:37.245446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:24:37.245446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:24:37.245543 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:24:37.245791 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:24:37.245795 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:24:37.245908 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 00:24:37.245990 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 00:24:37.246107 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":107,"threads_started":1}
I20260924 00:24:37.246779 16741 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37621
I20260924 00:24:37.246884 16741 heartbeater.cc:464] Registering TS with master...
I20260924 00:24:37.247042 16741 heartbeater.cc:511] Master 127.0.98.62:37621 requested a full tablet report, sending...
I20260924 00:24:37.247490 16282 ts_manager.cc:194] Registered new tserver with Master: 62d5fa98042b467a83ba0b2594e3ba4e (127.0.98.3:42597)
I20260924 00:24:37.247920 16282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41803
I20260924 00:24:37.250397 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:24:37.259050 16322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:37.251766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42946 (local address 127.0.98.62:37621)
0924 00:24:37.251935 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:37.251937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:37.252038 (+ 101us) server_negotiation.cc:407] Connection header received
0924 00:24:37.252126 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:37.252127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:37.252137 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:37.252267 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:37.252883 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.253427 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.254400 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.254613 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.254707 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:37.256073 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:37.256074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:37.256075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:37.256082 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:37.257737 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:37.258161 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:37.258163 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:37.258163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:37.258231 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:37.258450 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:37.258451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:37.258451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:37.258624 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:37.258743 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:37.258852 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 00:24:37.258935 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260924 00:24:37.260701 16282 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42946:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:24:37.262326 16282 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:24:37.269133 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:37.265091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33145 (local address 127.0.0.1:57942)
0924 00:24:37.265251 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:37.265292 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:37.265360 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:37.265975 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:37.265976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:37.265978 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:37.266052 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:37.266053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.267054 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.267055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:37.268042 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:37.268044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.268811 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.268812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:37.268899 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:37.268914 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 00:24:37.268985 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 00:24:37.269038 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":7,"thread_start_us":83,"threads_started":1}
I20260924 00:24:37.269133 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:37.265302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57942 (local address 127.0.98.2:33145)
0924 00:24:37.265818 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:37.265822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:37.265830 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:24:37.265866 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:37.265868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:37.265884 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:37.265963 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:37.266132 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.266956 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.268126 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.268710 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.268810 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:37.268870 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:37.269010 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 00:24:37.269047 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":84,"threads_started":1}
I20260924 00:24:37.269992 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:37.265126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41677 (local address 127.0.0.1:56760)
0924 00:24:37.266071 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:37.266081 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:37.266130 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:37.266347 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:37.266347 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:37.266349 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:37.266393 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:37.266395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.267606 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.267607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:37.268712 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:37.268714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.269742 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.269743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:37.269807 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:37.269811 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:24:37.269878 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 00:24:37.269917 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"thread_start_us":97,"threads_started":1}
I20260924 00:24:37.270005 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:37.265380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56760 (local address 127.0.98.1:41677)
0924 00:24:37.265558 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:37.265564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:37.266190 (+ 626us) server_negotiation.cc:407] Connection header received
0924 00:24:37.266239 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:37.266242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:37.266258 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:37.266334 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:37.266463 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.267493 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.268791 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.269637 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.269736 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:37.269796 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:37.269892 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 00:24:37.269929 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260924 00:24:37.269924 16541 tablet_service.cc:1467] Processing CreateTablet for tablet c65563a217b74cdfbcf5e520bf85c8d1 (DEFAULT_TABLE table=test-table [id=6be310588d7f4490be14b1026aa26f7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:24:37.270325 16541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c65563a217b74cdfbcf5e520bf85c8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:37.270682 16407 tablet_service.cc:1467] Processing CreateTablet for tablet c65563a217b74cdfbcf5e520bf85c8d1 (DEFAULT_TABLE table=test-table [id=6be310588d7f4490be14b1026aa26f7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:24:37.270970 16407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c65563a217b74cdfbcf5e520bf85c8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:37.272871 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:37.265091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42597 (local address 127.0.0.1:57000)
0924 00:24:37.265251 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:37.265292 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:37.265360 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:37.265965 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:37.265968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:37.265971 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:37.266058 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:37.266060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.267366 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.267367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:37.268601 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:37.268603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.272627 (+ 4024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.272628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:37.272695 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:37.272699 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:24:37.272759 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 00:24:37.272801 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":33,"thread_start_us":51,"threads_started":1}
I20260924 00:24:37.272890 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:37.265481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57000 (local address 127.0.98.3:42597)
0924 00:24:37.265678 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:37.265684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:37.265696 (+ 12us) server_negotiation.cc:407] Connection header received
0924 00:24:37.265753 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:37.265756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:37.265774 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:37.265867 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:37.266163 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.267254 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.268691 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.272528 (+ 3837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.272625 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:37.272682 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:37.272773 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 00:24:37.272814 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":205,"threads_started":1}
I20260924 00:24:37.273733 16758 tablet_bootstrap.cc:492] T c65563a217b74cdfbcf5e520bf85c8d1 P d8e56d382e9245ba8f563b2dc455552f: Bootstrap starting.
I20260924 00:24:37.274212 16759 tablet_bootstrap.cc:492] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e: Bootstrap starting.
I20260924 00:24:37.274413 16758 tablet_bootstrap.cc:654] T c65563a217b74cdfbcf5e520bf85c8d1 P d8e56d382e9245ba8f563b2dc455552f: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:37.274865 16759 tablet_bootstrap.cc:654] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:37.274859 16675 tablet_service.cc:1467] Processing CreateTablet for tablet c65563a217b74cdfbcf5e520bf85c8d1 (DEFAULT_TABLE table=test-table [id=6be310588d7f4490be14b1026aa26f7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:24:37.274971 16758 log.cc:824] T c65563a217b74cdfbcf5e520bf85c8d1 P d8e56d382e9245ba8f563b2dc455552f: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:37.275206 16675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c65563a217b74cdfbcf5e520bf85c8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:37.275409 16759 log.cc:824] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:37.275799 16758 tablet_bootstrap.cc:492] T c65563a217b74cdfbcf5e520bf85c8d1 P d8e56d382e9245ba8f563b2dc455552f: No bootstrap required, opened a new log
I20260924 00:24:37.275880 16758 ts_tablet_manager.cc:1397] T c65563a217b74cdfbcf5e520bf85c8d1 P d8e56d382e9245ba8f563b2dc455552f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:24:37.276295 16759 tablet_bootstrap.cc:492] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e: No bootstrap required, opened a new log
I20260924 00:24:37.276369 16759 ts_tablet_manager.cc:1397] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:24:37.277446 16758 raft_consensus.cc:348] T c65563a217b74cdfbcf5e520bf85c8d1 P d8e56d382e9245ba8f563b2dc455552f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e56d382e9245ba8f563b2dc455552f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33145 } } peers { permanent_uuid: "62d5fa98042b467a83ba0b2594e3ba4e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42597 } } peers { permanent_uuid: "f36e27af724642b0a971c16ce110b39e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41677 } }
I20260924 00:24:37.277635 16758 raft_consensus.cc:374] T c65563a217b74cdfbcf5e520bf85c8d1 P d8e56d382e9245ba8f563b2dc455552f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:37.277683 16758 raft_consensus.cc:729] T c65563a217b74cdfbcf5e520bf85c8d1 P d8e56d382e9245ba8f563b2dc455552f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8e56d382e9245ba8f563b2dc455552f, State: Initialized, Role: FOLLOWER
I20260924 00:24:37.277813 16758 consensus_queue.cc:260] T c65563a217b74cdfbcf5e520bf85c8d1 P d8e56d382e9245ba8f563b2dc455552f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e56d382e9245ba8f563b2dc455552f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33145 } } peers { permanent_uuid: "62d5fa98042b467a83ba0b2594e3ba4e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42597 } } peers { permanent_uuid: "f36e27af724642b0a971c16ce110b39e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41677 } }
I20260924 00:24:37.278105 16607 heartbeater.cc:503] Master 127.0.98.62:37621 was elected leader, sending a full tablet report...
I20260924 00:24:37.278131 16758 ts_tablet_manager.cc:1428] T c65563a217b74cdfbcf5e520bf85c8d1 P d8e56d382e9245ba8f563b2dc455552f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 00:24:37.278539 16763 tablet_bootstrap.cc:492] T c65563a217b74cdfbcf5e520bf85c8d1 P 62d5fa98042b467a83ba0b2594e3ba4e: Bootstrap starting.
I20260924 00:24:37.278441 16759 raft_consensus.cc:348] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e56d382e9245ba8f563b2dc455552f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33145 } } peers { permanent_uuid: "62d5fa98042b467a83ba0b2594e3ba4e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42597 } } peers { permanent_uuid: "f36e27af724642b0a971c16ce110b39e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41677 } }
I20260924 00:24:37.278612 16759 raft_consensus.cc:374] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:37.278659 16759 raft_consensus.cc:729] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36e27af724642b0a971c16ce110b39e, State: Initialized, Role: FOLLOWER
I20260924 00:24:37.278784 16759 consensus_queue.cc:260] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e56d382e9245ba8f563b2dc455552f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33145 } } peers { permanent_uuid: "62d5fa98042b467a83ba0b2594e3ba4e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42597 } } peers { permanent_uuid: "f36e27af724642b0a971c16ce110b39e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41677 } }
I20260924 00:24:37.279074 16759 ts_tablet_manager.cc:1428] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:24:37.279124 16473 heartbeater.cc:503] Master 127.0.98.62:37621 was elected leader, sending a full tablet report...
I20260924 00:24:37.279496 16763 tablet_bootstrap.cc:654] T c65563a217b74cdfbcf5e520bf85c8d1 P 62d5fa98042b467a83ba0b2594e3ba4e: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:37.280216 16763 log.cc:824] T c65563a217b74cdfbcf5e520bf85c8d1 P 62d5fa98042b467a83ba0b2594e3ba4e: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:37.280824 16763 tablet_bootstrap.cc:492] T c65563a217b74cdfbcf5e520bf85c8d1 P 62d5fa98042b467a83ba0b2594e3ba4e: No bootstrap required, opened a new log
I20260924 00:24:37.280875 16763 ts_tablet_manager.cc:1397] T c65563a217b74cdfbcf5e520bf85c8d1 P 62d5fa98042b467a83ba0b2594e3ba4e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:24:37.282133 16763 raft_consensus.cc:348] T c65563a217b74cdfbcf5e520bf85c8d1 P 62d5fa98042b467a83ba0b2594e3ba4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e56d382e9245ba8f563b2dc455552f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33145 } } peers { permanent_uuid: "62d5fa98042b467a83ba0b2594e3ba4e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42597 } } peers { permanent_uuid: "f36e27af724642b0a971c16ce110b39e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41677 } }
I20260924 00:24:37.282325 16763 raft_consensus.cc:374] T c65563a217b74cdfbcf5e520bf85c8d1 P 62d5fa98042b467a83ba0b2594e3ba4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:37.282388 16763 raft_consensus.cc:729] T c65563a217b74cdfbcf5e520bf85c8d1 P 62d5fa98042b467a83ba0b2594e3ba4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62d5fa98042b467a83ba0b2594e3ba4e, State: Initialized, Role: FOLLOWER
I20260924 00:24:37.282517 16763 consensus_queue.cc:260] T c65563a217b74cdfbcf5e520bf85c8d1 P 62d5fa98042b467a83ba0b2594e3ba4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e56d382e9245ba8f563b2dc455552f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33145 } } peers { permanent_uuid: "62d5fa98042b467a83ba0b2594e3ba4e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42597 } } peers { permanent_uuid: "f36e27af724642b0a971c16ce110b39e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41677 } }
I20260924 00:24:37.282817 16763 ts_tablet_manager.cc:1428] T c65563a217b74cdfbcf5e520bf85c8d1 P 62d5fa98042b467a83ba0b2594e3ba4e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:24:37.282829 16741 heartbeater.cc:503] Master 127.0.98.62:37621 was elected leader, sending a full tablet report...
W20260924 00:24:37.320289 16608 tablet.cc:2369] T c65563a217b74cdfbcf5e520bf85c8d1 P d8e56d382e9245ba8f563b2dc455552f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:24:37.424022 16474 tablet.cc:2369] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 00:24:37.486465 16742 tablet.cc:2369] T c65563a217b74cdfbcf5e520bf85c8d1 P 62d5fa98042b467a83ba0b2594e3ba4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:24:37.551713 16764 raft_consensus.cc:482] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:24:37.551855 16764 raft_consensus.cc:504] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e56d382e9245ba8f563b2dc455552f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33145 } } peers { permanent_uuid: "62d5fa98042b467a83ba0b2594e3ba4e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42597 } } peers { permanent_uuid: "f36e27af724642b0a971c16ce110b39e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41677 } }
I20260924 00:24:37.552433 16764 leader_election.cc:288] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8e56d382e9245ba8f563b2dc455552f (127.0.98.2:33145), 62d5fa98042b467a83ba0b2594e3ba4e (127.0.98.3:42597)
I20260924 00:24:37.556006 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:37.552754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45159 (local address 127.0.98.2:33145)
0924 00:24:37.552807 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:37.552809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:37.552867 (+ 58us) server_negotiation.cc:407] Connection header received
0924 00:24:37.552959 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:37.552960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:37.552973 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:37.553056 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:37.553342 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.553851 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.555281 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.555638 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.555725 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:37.555735 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:37.555898 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 00:24:37.555933 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 00:24:37.556021 16767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:37.552663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33145 (local address 127.0.98.1:45159)
0924 00:24:37.552803 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:37.552816 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:37.552888 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:37.553116 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:37.553117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:37.553120 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:37.553247 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:37.553250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.553956 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.553957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:37.555117 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:37.555120 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.555727 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.555728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:37.555820 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:37.555824 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:24:37.555891 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 00:24:37.555933 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260924 00:24:37.556295 16768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:37.552663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42597 (local address 127.0.98.1:59549)
0924 00:24:37.552860 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:37.552869 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:37.552925 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:37.553084 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:37.553085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:37.553088 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:37.553173 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:37.553175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.554012 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.554013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:37.555099 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:37.555102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.555918 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.555919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:37.556008 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:37.556012 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:24:37.556093 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 00:24:37.556149 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":2,"thread_start_us":61,"threads_started":1}
I20260924 00:24:37.556459 16561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65563a217b74cdfbcf5e520bf85c8d1" candidate_uuid: "f36e27af724642b0a971c16ce110b39e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8e56d382e9245ba8f563b2dc455552f" is_pre_election: true
I20260924 00:24:37.556666 16561 raft_consensus.cc:2454] T c65563a217b74cdfbcf5e520bf85c8d1 P d8e56d382e9245ba8f563b2dc455552f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f36e27af724642b0a971c16ce110b39e in term 0.
I20260924 00:24:37.556715 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:37.552754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59549 (local address 127.0.98.3:42597)
0924 00:24:37.552821 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:37.552823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:37.552948 (+ 125us) server_negotiation.cc:407] Connection header received
0924 00:24:37.552979 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:37.552981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:37.552991 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:37.553055 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:37.553257 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.553932 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.555206 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.555824 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.555913 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:37.555923 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:37.556580 (+ 657us) server_negotiation.cc:299] Negotiation successful
0924 00:24:37.556642 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 00:24:37.556967 16363 leader_election.cc:302] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8e56d382e9245ba8f563b2dc455552f, f36e27af724642b0a971c16ce110b39e; no voters:
I20260924 00:24:37.557017 16695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65563a217b74cdfbcf5e520bf85c8d1" candidate_uuid: "f36e27af724642b0a971c16ce110b39e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62d5fa98042b467a83ba0b2594e3ba4e" is_pre_election: true
I20260924 00:24:37.557139 16764 raft_consensus.cc:2781] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:24:37.557179 16695 raft_consensus.cc:2454] T c65563a217b74cdfbcf5e520bf85c8d1 P 62d5fa98042b467a83ba0b2594e3ba4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f36e27af724642b0a971c16ce110b39e in term 0.
I20260924 00:24:37.557204 16764 raft_consensus.cc:482] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:24:37.557264 16764 raft_consensus.cc:3037] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:37.558050 16764 raft_consensus.cc:504] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e56d382e9245ba8f563b2dc455552f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33145 } } peers { permanent_uuid: "62d5fa98042b467a83ba0b2594e3ba4e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42597 } } peers { permanent_uuid: "f36e27af724642b0a971c16ce110b39e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41677 } }
I20260924 00:24:37.558280 16764 leader_election.cc:288] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e [CANDIDATE]: Term 1 election: Requested vote from peers d8e56d382e9245ba8f563b2dc455552f (127.0.98.2:33145), 62d5fa98042b467a83ba0b2594e3ba4e (127.0.98.3:42597)
I20260924 00:24:37.558349 16561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65563a217b74cdfbcf5e520bf85c8d1" candidate_uuid: "f36e27af724642b0a971c16ce110b39e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8e56d382e9245ba8f563b2dc455552f"
I20260924 00:24:37.558454 16561 raft_consensus.cc:3037] T c65563a217b74cdfbcf5e520bf85c8d1 P d8e56d382e9245ba8f563b2dc455552f [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:37.558440 16695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65563a217b74cdfbcf5e520bf85c8d1" candidate_uuid: "f36e27af724642b0a971c16ce110b39e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62d5fa98042b467a83ba0b2594e3ba4e"
I20260924 00:24:37.558547 16695 raft_consensus.cc:3037] T c65563a217b74cdfbcf5e520bf85c8d1 P 62d5fa98042b467a83ba0b2594e3ba4e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:37.559235 16561 raft_consensus.cc:2454] T c65563a217b74cdfbcf5e520bf85c8d1 P d8e56d382e9245ba8f563b2dc455552f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f36e27af724642b0a971c16ce110b39e in term 1.
I20260924 00:24:37.559362 16695 raft_consensus.cc:2454] T c65563a217b74cdfbcf5e520bf85c8d1 P 62d5fa98042b467a83ba0b2594e3ba4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f36e27af724642b0a971c16ce110b39e in term 1.
I20260924 00:24:37.559406 16363 leader_election.cc:302] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8e56d382e9245ba8f563b2dc455552f, f36e27af724642b0a971c16ce110b39e; no voters:
I20260924 00:24:37.559558 16764 raft_consensus.cc:2781] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:37.559733 16764 raft_consensus.cc:686] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e [term 1 LEADER]: Becoming Leader. State: Replica: f36e27af724642b0a971c16ce110b39e, State: Running, Role: LEADER
I20260924 00:24:37.559859 16764 consensus_queue.cc:237] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e56d382e9245ba8f563b2dc455552f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33145 } } peers { permanent_uuid: "62d5fa98042b467a83ba0b2594e3ba4e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42597 } } peers { permanent_uuid: "f36e27af724642b0a971c16ce110b39e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41677 } }
I20260924 00:24:37.560844 16282 catalog_manager.cc:5495] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e reported cstate change: term changed from 0 to 1, leader changed from <none> to f36e27af724642b0a971c16ce110b39e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f36e27af724642b0a971c16ce110b39e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8e56d382e9245ba8f563b2dc455552f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62d5fa98042b467a83ba0b2594e3ba4e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f36e27af724642b0a971c16ce110b39e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41677 } health_report { overall_health: HEALTHY } } }
I20260924 00:24:37.629329 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:37.626609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56770 (local address 127.0.98.1:41677)
0924 00:24:37.626753 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:37.626755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:37.626765 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:24:37.626807 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:37.626808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:37.626830 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:37.626918 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:24:37.627100 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.627624 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.628499 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.628770 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.628834 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:37.628979 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:24:37.629094 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:24:37.629169 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:37.629187 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 00:24:37.629204 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260924 00:24:37.631464 16695 raft_consensus.cc:1261] T c65563a217b74cdfbcf5e520bf85c8d1 P 62d5fa98042b467a83ba0b2594e3ba4e [term 1 FOLLOWER]: Refusing update from remote peer f36e27af724642b0a971c16ce110b39e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:24:37.631505 16561 raft_consensus.cc:1261] T c65563a217b74cdfbcf5e520bf85c8d1 P d8e56d382e9245ba8f563b2dc455552f [term 1 FOLLOWER]: Refusing update from remote peer f36e27af724642b0a971c16ce110b39e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:24:37.631863 16769 consensus_queue.cc:1035] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "62d5fa98042b467a83ba0b2594e3ba4e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:24:37.632081 16764 consensus_queue.cc:1035] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8e56d382e9245ba8f563b2dc455552f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:24:37.636945 16281 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:24:37.637063 16281 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42946:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:24:37.637888 16281 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53c9ed0ccc234ff4a0b231ff82720c3f: Sending DeleteTablet for 3 replicas of tablet c65563a217b74cdfbcf5e520bf85c8d1
I20260924 00:24:37.638265 16407 tablet_service.cc:1514] Processing DeleteTablet for tablet c65563a217b74cdfbcf5e520bf85c8d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:24:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56760
I20260924 00:24:37.638312 16541 tablet_service.cc:1514] Processing DeleteTablet for tablet c65563a217b74cdfbcf5e520bf85c8d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:24:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57942
I20260924 00:24:37.638528 16782 tablet_replica.cc:330] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e: stopping tablet replica
I20260924 00:24:37.638649 16782 raft_consensus.cc:2229] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:24:37.638659 16675 tablet_service.cc:1514] Processing DeleteTablet for tablet c65563a217b74cdfbcf5e520bf85c8d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:24:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57000
I20260924 00:24:37.638769 16783 tablet_replica.cc:330] T c65563a217b74cdfbcf5e520bf85c8d1 P d8e56d382e9245ba8f563b2dc455552f: stopping tablet replica
I20260924 00:24:37.638870 16783 raft_consensus.cc:2229] T c65563a217b74cdfbcf5e520bf85c8d1 P d8e56d382e9245ba8f563b2dc455552f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:24:37.638892 16782 raft_consensus.cc:2258] T c65563a217b74cdfbcf5e520bf85c8d1 P f36e27af724642b0a971c16ce110b39e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:24:37.638976 16783 raft_consensus.cc:2258] T c65563a217b74cdfbcf5e520bf85c8d1 P d8e56d382e9245ba8f563b2dc455552f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:24:37.639140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 16344
I20260924 00:24:37.640132 16784 tablet_replica.cc:330] T c65563a217b74cdfbcf5e520bf85c8d1 P 62d5fa98042b467a83ba0b2594e3ba4e: stopping tablet replica
I20260924 00:24:37.640276 16784 raft_consensus.cc:2229] T c65563a217b74cdfbcf5e520bf85c8d1 P 62d5fa98042b467a83ba0b2594e3ba4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:24:37.640380 16784 raft_consensus.cc:2258] T c65563a217b74cdfbcf5e520bf85c8d1 P 62d5fa98042b467a83ba0b2594e3ba4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:24:37.640825 16784 ts_tablet_manager.cc:1905] T c65563a217b74cdfbcf5e520bf85c8d1 P 62d5fa98042b467a83ba0b2594e3ba4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:24:37.643121 16784 ts_tablet_manager.cc:1918] T c65563a217b74cdfbcf5e520bf85c8d1 P 62d5fa98042b467a83ba0b2594e3ba4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:24:37.643185 16784 log.cc:1196] T c65563a217b74cdfbcf5e520bf85c8d1 P 62d5fa98042b467a83ba0b2594e3ba4e: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/c65563a217b74cdfbcf5e520bf85c8d1
I20260924 00:24:37.643469 16784 ts_tablet_manager.cc:1939] T c65563a217b74cdfbcf5e520bf85c8d1 P 62d5fa98042b467a83ba0b2594e3ba4e: Deleting consensus metadata
I20260924 00:24:37.643947 16267 catalog_manager.cc:4841] TS 62d5fa98042b467a83ba0b2594e3ba4e (127.0.98.3:42597): tablet c65563a217b74cdfbcf5e520bf85c8d1 (table test-table [id=6be310588d7f4490be14b1026aa26f7a]) successfully deleted
I20260924 00:24:37.644554 16783 ts_tablet_manager.cc:1905] T c65563a217b74cdfbcf5e520bf85c8d1 P d8e56d382e9245ba8f563b2dc455552f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 00:24:37.645424 16268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41677 (error 108)
I20260924 00:24:37.645704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 16478
I20260924 00:24:37.646536 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:37.646386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41677 (local address 127.0.0.1:56780)
0924 00:24:37.646451 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:37.646496 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 00:24:37.646672 16268 catalog_manager.cc:4584] TS f36e27af724642b0a971c16ce110b39e (127.0.98.1:41677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c65563a217b74cdfbcf5e520bf85c8d1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41677: connect: Connection refused (error 111)
I20260924 00:24:37.651319 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 16612
I20260924 00:24:37.651450 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:37.651313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33145 (local address 127.0.0.1:57950)
0924 00:24:37.651376 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:37.651407 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 00:24:37.655874 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 16250
2026-09-24T00:24:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2298 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T00:24:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:24:37Z Disabled control of system clock
I20260924 00:24:37.683375 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43103
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 00:24:37.766055 16789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:37.766338 16789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:37.766386 16789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:37.767771 16789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:24:37.767824 16789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:37.767838 16789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:37.767850 16789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:24:37.767892 16789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:24:37.769327 16789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43103
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:37.769596 16789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:37.769905 16789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:37.775074 16795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:37.775091 16797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:37.775177 16789 server_base.cc:1034] running on GCE node
W20260924 00:24:37.775063 16794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:37.775524 16789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:37.775822 16789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:37.776989 16789 hybrid_clock.cc:648] HybridClock initialized: now 1790209477776976 us; error 45 us; skew 500 ppm
I20260924 00:24:37.778399 16789 webserver.cc:460] Webserver started at http://127.0.98.62:43665/ using document root <none> and password file <none>
I20260924 00:24:37.778647 16789 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:37.778719 16789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:37.778908 16789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:37.780046 16789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "7d37336847b74d99b5c028076c2d7c7d"
format_stamp: "Formatted at 2026-09-24 00:24:37 on dist-test-slave-777x"
server_key: "ab386e87d7d1974b4cc7cbda4512b0c0"
server_key_iv: "1eeb08934eed19c9c636c5f63dd5e738"
server_key_version: "encryptionkey@0"
I20260924 00:24:37.780467 16789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "7d37336847b74d99b5c028076c2d7c7d"
format_stamp: "Formatted at 2026-09-24 00:24:37 on dist-test-slave-777x"
server_key: "ab386e87d7d1974b4cc7cbda4512b0c0"
server_key_iv: "1eeb08934eed19c9c636c5f63dd5e738"
server_key_version: "encryptionkey@0"
I20260924 00:24:37.781839 16789 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:37.782562 16803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:37.782728 16789 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:24:37.782855 16789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "7d37336847b74d99b5c028076c2d7c7d"
format_stamp: "Formatted at 2026-09-24 00:24:37 on dist-test-slave-777x"
server_key: "ab386e87d7d1974b4cc7cbda4512b0c0"
server_key_iv: "1eeb08934eed19c9c636c5f63dd5e738"
server_key_version: "encryptionkey@0"
I20260924 00:24:37.782932 16789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:37.816941 16789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:37.817304 16789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:37.817471 16789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:37.821538 16789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43103
I20260924 00:24:37.821576 16855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43103 every 8 connection(s)
I20260924 00:24:37.821954 16789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:24:37.822922 16856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:37.825178 16856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d: Bootstrap starting.
I20260924 00:24:37.825711 16856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:37.825953 16856 log.cc:824] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:37.826596 16856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d: No bootstrap required, opened a new log
I20260924 00:24:37.828096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 16789
I20260924 00:24:37.828234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:24:37.828181 16856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d37336847b74d99b5c028076c2d7c7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43103 } }
I20260924 00:24:37.828411 16856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:37.828481 392 external_mini_cluster.cc:1428] Setting key 811244adfdfbbd6166ede1f06f389aea
I20260924 00:24:37.828478 16856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d37336847b74d99b5c028076c2d7c7d, State: Initialized, Role: FOLLOWER
I20260924 00:24:37.828624 16856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d37336847b74d99b5c028076c2d7c7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43103 } }
I20260924 00:24:37.828713 16856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:24:37.828750 16856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:24:37.828802 16856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:37.829509 16856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d37336847b74d99b5c028076c2d7c7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43103 } }
I20260924 00:24:37.829618 16856 leader_election.cc:302] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d37336847b74d99b5c028076c2d7c7d; no voters:
I20260924 00:24:37.829779 16856 leader_election.cc:288] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:24:37.829831 16861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:37.829975 16861 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d [term 1 LEADER]: Becoming Leader. State: Replica: 7d37336847b74d99b5c028076c2d7c7d, State: Running, Role: LEADER
I20260924 00:24:37.830032 16856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:24:37.830071 16861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d37336847b74d99b5c028076c2d7c7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43103 } }
I20260924 00:24:37.830449 16861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d37336847b74d99b5c028076c2d7c7d. Latest consensus state: current_term: 1 leader_uuid: "7d37336847b74d99b5c028076c2d7c7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d37336847b74d99b5c028076c2d7c7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43103 } } }
I20260924 00:24:37.830524 16861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:37.830662 16863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d37336847b74d99b5c028076c2d7c7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d37336847b74d99b5c028076c2d7c7d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43103 } } }
I20260924 00:24:37.830715 16863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:37.830813 16868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:24:37.831291 16868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:24:37.833003 16868 catalog_manager.cc:1347] Generated new cluster ID: 62363feeb9a54a65beb1e4827ad220df
I20260924 00:24:37.833057 16868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:24:37.835297 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:37.829031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59814 (local address 127.0.98.62:43103)
0924 00:24:37.829283 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:37.829294 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:37.829335 (+ 41us) server_negotiation.cc:407] Connection header received
0924 00:24:37.829465 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:37.829473 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:37.829558 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:37.829720 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:37.830070 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.831837 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.832966 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.833538 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.833647 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:37.834544 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:37.834547 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:37.834548 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:37.834738 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:37.834784 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:37.834797 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:37.834845 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:37.835019 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 00:24:37.835092 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":115,"threads_started":1}
I20260924 00:24:37.840520 16868 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:24:37.841190 16868 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:24:37.846576 16868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d: Generated new TSK 0
I20260924 00:24:37.846753 16868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:24:37.850307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43103
--builtin_ntp_servers=127.0.98.20:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 00:24:37.933281 16880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:37.933526 16880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:37.933565 16880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:37.934801 16880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:37.934860 16880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:37.934902 16880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:24:37.936340 16880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:37.936595 16880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:37.936914 16880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:37.942131 16886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:37.942101 16885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:37.942107 16888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:37.942462 16880 server_base.cc:1034] running on GCE node
I20260924 00:24:37.942667 16880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:37.942950 16880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:37.944105 16880 hybrid_clock.cc:648] HybridClock initialized: now 1790209477944086 us; error 35 us; skew 500 ppm
I20260924 00:24:37.945577 16880 webserver.cc:460] Webserver started at http://127.0.98.1:39613/ using document root <none> and password file <none>
I20260924 00:24:37.945852 16880 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:37.945924 16880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:37.946080 16880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:37.947171 16880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "39a97cd362154ffeb98badca8f1c887f"
format_stamp: "Formatted at 2026-09-24 00:24:37 on dist-test-slave-777x"
server_key: "9cde65e23d84dc78fe06eb5eb9b14562"
server_key_iv: "a97707c74e467f996506dd4ef80e74f8"
server_key_version: "encryptionkey@0"
I20260924 00:24:37.947537 16880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "39a97cd362154ffeb98badca8f1c887f"
format_stamp: "Formatted at 2026-09-24 00:24:37 on dist-test-slave-777x"
server_key: "9cde65e23d84dc78fe06eb5eb9b14562"
server_key_iv: "a97707c74e467f996506dd4ef80e74f8"
server_key_version: "encryptionkey@0"
I20260924 00:24:37.948980 16880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:24:37.949759 16894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:37.949932 16880 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:37.950035 16880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "39a97cd362154ffeb98badca8f1c887f"
format_stamp: "Formatted at 2026-09-24 00:24:37 on dist-test-slave-777x"
server_key: "9cde65e23d84dc78fe06eb5eb9b14562"
server_key_iv: "a97707c74e467f996506dd4ef80e74f8"
server_key_version: "encryptionkey@0"
I20260924 00:24:37.950116 16880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:37.969226 16880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:37.969561 16880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:37.969684 16880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:37.970006 16880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:24:37.970358 16880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:24:37.970394 16880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:37.970439 16880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:24:37.970465 16880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:37.976761 16880 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46043
I20260924 00:24:37.976811 17007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46043 every 8 connection(s)
I20260924 00:24:37.977240 16880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:24:37.985101 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:37.978180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46425 (local address 127.0.98.62:43103)
0924 00:24:37.978231 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:37.978233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:37.978613 (+ 380us) server_negotiation.cc:407] Connection header received
0924 00:24:37.979513 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:37.979514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:37.979532 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:37.979601 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:37.980725 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.981654 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.983227 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.983441 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.983502 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:37.984740 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:37.984742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:37.984742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:37.984747 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:37.984777 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:37.984782 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:37.984828 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:37.984946 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 00:24:37.984982 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:24:37.985585 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 16880
I20260924 00:24:37.985728 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:24:37.985921 392 external_mini_cluster.cc:1428] Setting key b6f44fc817aef652d42cc174939b6f48
I20260924 00:24:37.986224 17010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:37.978154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43103 (local address 127.0.98.1:46425)
0924 00:24:37.978414 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:37.978488 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:37.979380 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:37.979708 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:37.979713 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:37.979726 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:24:37.980588 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:37.980597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.981777 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.981779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:37.983150 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:37.983152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:37.983531 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:37.983532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:37.983775 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:37.984557 (+ 782us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:24:37.984558 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:24:37.984591 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:24:37.984646 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:24:37.984834 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:24:37.984841 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:24:37.984844 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:24:37.984931 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 00:24:37.985052 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":107,"threads_started":1}
I20260924 00:24:37.986996 17008 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43103
I20260924 00:24:37.987113 17008 heartbeater.cc:464] Registering TS with master...
I20260924 00:24:37.987216 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43103
--builtin_ntp_servers=127.0.98.20:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 00:24:37.987367 17008 heartbeater.cc:511] Master 127.0.98.62:43103 requested a full tablet report, sending...
I20260924 00:24:37.987855 16820 ts_manager.cc:194] Registered new tserver with Master: 39a97cd362154ffeb98badca8f1c887f (127.0.98.1:46043)
I20260924 00:24:37.988559 16820 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:46425
W20260924 00:24:38.068622 17011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:38.068830 17011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:38.068852 17011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:38.069999 17011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:38.070039 17011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:38.070065 17011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:24:38.071427 17011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:38.071679 17011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:38.071992 17011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:38.076962 17019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:38.076970 17017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:38.076970 17016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:38.077486 17011 server_base.cc:1034] running on GCE node
I20260924 00:24:38.077711 17011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:38.077968 17011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:38.079131 17011 hybrid_clock.cc:648] HybridClock initialized: now 1790209478079124 us; error 40 us; skew 500 ppm
I20260924 00:24:38.080538 17011 webserver.cc:460] Webserver started at http://127.0.98.2:37547/ using document root <none> and password file <none>
I20260924 00:24:38.080780 17011 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:38.080848 17011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:38.081007 17011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:38.082099 17011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "f6d2e5e314aa48868c4735f12a8b4f26"
format_stamp: "Formatted at 2026-09-24 00:24:38 on dist-test-slave-777x"
server_key: "9f44c5e27fb637cbdc79dc2ad7cea3a9"
server_key_iv: "9bae5958c83786124ee9e69426dd91f7"
server_key_version: "encryptionkey@0"
I20260924 00:24:38.082472 17011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f6d2e5e314aa48868c4735f12a8b4f26"
format_stamp: "Formatted at 2026-09-24 00:24:38 on dist-test-slave-777x"
server_key: "9f44c5e27fb637cbdc79dc2ad7cea3a9"
server_key_iv: "9bae5958c83786124ee9e69426dd91f7"
server_key_version: "encryptionkey@0"
I20260924 00:24:38.083801 17011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:24:38.084616 17025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:38.084793 17011 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:38.084887 17011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "f6d2e5e314aa48868c4735f12a8b4f26"
format_stamp: "Formatted at 2026-09-24 00:24:38 on dist-test-slave-777x"
server_key: "9f44c5e27fb637cbdc79dc2ad7cea3a9"
server_key_iv: "9bae5958c83786124ee9e69426dd91f7"
server_key_version: "encryptionkey@0"
I20260924 00:24:38.084969 17011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:38.108994 17011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:38.109393 17011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:38.109579 17011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:38.109862 17011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:24:38.110234 17011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:24:38.110296 17011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:38.110340 17011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:24:38.110373 17011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:38.116847 17011 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38157
I20260924 00:24:38.116930 17138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38157 every 8 connection(s)
I20260924 00:24:38.117285 17011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:24:38.122440 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:38.118077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48293 (local address 127.0.98.62:43103)
0924 00:24:38.118146 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:38.118148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:38.118370 (+ 222us) server_negotiation.cc:407] Connection header received
0924 00:24:38.118851 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:38.118852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:38.118862 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:38.118930 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:38.119566 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.120067 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.121159 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.121349 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.121407 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:38.122199 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:38.122200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:38.122200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:38.122205 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:38.122219 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:38.122223 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:38.122272 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:38.122354 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 00:24:38.122379 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 00:24:38.122648 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:38.118060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43103 (local address 127.0.98.2:48293)
0924 00:24:38.118237 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:38.118278 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:38.118761 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:38.118985 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:38.118988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:38.118996 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:24:38.119485 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:38.119489 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.120192 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.120194 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:38.121040 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:38.121042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.121425 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.121426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:38.121530 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:38.122062 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:24:38.122062 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:24:38.122091 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:24:38.122112 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:24:38.122281 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:24:38.122285 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:24:38.122287 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:24:38.122339 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 00:24:38.122416 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260924 00:24:38.122857 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 17011
I20260924 00:24:38.122967 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:24:38.123147 17139 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43103
I20260924 00:24:38.123222 392 external_mini_cluster.cc:1428] Setting key b56eefc8559c1de1f653f600fde48983
I20260924 00:24:38.123243 17139 heartbeater.cc:464] Registering TS with master...
I20260924 00:24:38.123461 17139 heartbeater.cc:511] Master 127.0.98.62:43103 requested a full tablet report, sending...
I20260924 00:24:38.123881 16820 ts_manager.cc:194] Registered new tserver with Master: f6d2e5e314aa48868c4735f12a8b4f26 (127.0.98.2:38157)
I20260924 00:24:38.124285 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43103
--builtin_ntp_servers=127.0.98.20:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 00:24:38.124331 16820 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:48293
W20260924 00:24:38.205507 17142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:38.205726 17142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:38.205749 17142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:38.206903 17142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:38.206944 17142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:38.206969 17142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:24:38.208374 17142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:38.208619 17142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:38.208940 17142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:38.214139 17148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:38.214128 17150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:38.214128 17147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:38.214572 17142 server_base.cc:1034] running on GCE node
I20260924 00:24:38.214790 17142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:38.214986 17142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:38.216159 17142 hybrid_clock.cc:648] HybridClock initialized: now 1790209478216143 us; error 27 us; skew 500 ppm
I20260924 00:24:38.217587 17142 webserver.cc:460] Webserver started at http://127.0.98.3:43613/ using document root <none> and password file <none>
I20260924 00:24:38.217872 17142 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:38.217947 17142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:38.218098 17142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:38.219229 17142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "6234c49eafc446b8953f7813a2d4e195"
format_stamp: "Formatted at 2026-09-24 00:24:38 on dist-test-slave-777x"
server_key: "45d79dad5e05c037732cd39b34927a02"
server_key_iv: "c7c9e2bcad346c7734c1b124b87b5b37"
server_key_version: "encryptionkey@0"
I20260924 00:24:38.219592 17142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6234c49eafc446b8953f7813a2d4e195"
format_stamp: "Formatted at 2026-09-24 00:24:38 on dist-test-slave-777x"
server_key: "45d79dad5e05c037732cd39b34927a02"
server_key_iv: "c7c9e2bcad346c7734c1b124b87b5b37"
server_key_version: "encryptionkey@0"
I20260924 00:24:38.220968 17142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:24:38.221665 17156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:38.221846 17142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:38.221942 17142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "6234c49eafc446b8953f7813a2d4e195"
format_stamp: "Formatted at 2026-09-24 00:24:38 on dist-test-slave-777x"
server_key: "45d79dad5e05c037732cd39b34927a02"
server_key_iv: "c7c9e2bcad346c7734c1b124b87b5b37"
server_key_version: "encryptionkey@0"
I20260924 00:24:38.222020 17142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:38.242338 17142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:38.242780 17142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:38.242959 17142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:38.243222 17142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:24:38.243594 17142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:24:38.243656 17142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:38.243716 17142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:24:38.243750 17142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:38.250077 17142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42813
I20260924 00:24:38.250121 17269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42813 every 8 connection(s)
I20260924 00:24:38.250471 17142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:24:38.256354 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:38.251293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43111 (local address 127.0.98.62:43103)
0924 00:24:38.251370 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:38.251372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:38.251660 (+ 288us) server_negotiation.cc:407] Connection header received
0924 00:24:38.252327 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:38.252329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:38.252340 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:38.252403 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:38.253130 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.253727 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.254803 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.254984 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.255035 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:38.255934 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:38.255936 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:38.255936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:38.255940 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:38.255955 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:38.255959 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:38.256006 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:38.256111 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 00:24:38.256139 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 00:24:38.256392 17272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:38.251251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43103 (local address 127.0.98.3:43111)
0924 00:24:38.251479 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:38.251526 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:38.252131 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:38.252468 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:38.252472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:38.252480 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:24:38.253039 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:38.253044 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.253824 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.253825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:38.254723 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:38.254725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.255086 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.255087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:38.255205 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:38.255800 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:24:38.255800 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:24:38.255827 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:24:38.255849 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:24:38.256016 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:24:38.256020 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:24:38.256022 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:24:38.256075 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 00:24:38.256152 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":124,"threads_started":1}
I20260924 00:24:38.256873 17270 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43103
I20260924 00:24:38.256978 17270 heartbeater.cc:464] Registering TS with master...
I20260924 00:24:38.257158 17270 heartbeater.cc:511] Master 127.0.98.62:43103 requested a full tablet report, sending...
I20260924 00:24:38.257534 16820 ts_manager.cc:194] Registered new tserver with Master: 6234c49eafc446b8953f7813a2d4e195 (127.0.98.3:42813)
I20260924 00:24:38.257920 16820 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:43111
I20260924 00:24:38.259307 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 17142
I20260924 00:24:38.259426 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:24:38.259636 392 external_mini_cluster.cc:1428] Setting key 6ffdb787742fea1d5906f9b11eb85028
I20260924 00:24:38.260476 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:24:38.264845 16860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:38.261825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59824 (local address 127.0.98.62:43103)
0924 00:24:38.261895 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:38.261897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:38.261986 (+ 89us) server_negotiation.cc:407] Connection header received
0924 00:24:38.262052 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:38.262053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:38.262066 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:38.262136 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:38.262323 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.262800 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.263565 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.263755 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.263831 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:38.264583 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:38.264585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:38.264585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:38.264590 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:38.264604 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:38.264608 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:38.264676 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:38.264739 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 00:24:38.264762 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 00:24:38.266428 16820 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:24:38.268138 16820 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:24:38.274744 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:38.270412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46043 (local address 127.0.0.1:60908)
0924 00:24:38.270539 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:38.270580 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:38.270643 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:38.270978 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:38.270982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:38.270985 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:38.271060 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:38.271062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.272395 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.272396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:38.273506 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:38.273509 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.274444 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.274445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:38.274521 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:38.274539 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 00:24:38.274605 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 00:24:38.274660 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260924 00:24:38.274873 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:38.270642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42813 (local address 127.0.0.1:51498)
0924 00:24:38.270768 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:38.270777 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:38.270819 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:38.271201 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:38.271202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:38.271203 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:38.271244 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:38.271245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.272521 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.272522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:38.273572 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:38.273575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.274600 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.274601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:38.274662 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:38.274667 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:24:38.274730 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 00:24:38.274783 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":88,"threads_started":1}
I20260924 00:24:38.274986 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:38.270869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38157 (local address 127.0.0.1:45738)
0924 00:24:38.271069 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:38.271079 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:38.271125 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:38.271662 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:38.271663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:38.271665 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:38.271700 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:38.271701 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.272974 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.272975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:38.273884 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:38.273886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.274686 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.274686 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:38.274742 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:38.274746 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:24:38.274797 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 00:24:38.274838 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":62,"threads_started":1}
I20260924 00:24:38.275045 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:38.271059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45738 (local address 127.0.98.2:38157)
0924 00:24:38.271422 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:38.271426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:38.271437 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:24:38.271474 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:38.271476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:38.271492 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:38.271567 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:38.271789 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.272887 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.273968 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.274493 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.274558 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:38.274652 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:38.274914 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 00:24:38.274958 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":97,"threads_started":1}
I20260924 00:24:38.275202 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:38.270528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60908 (local address 127.0.98.1:46043)
0924 00:24:38.270782 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:38.270788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:38.270799 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:24:38.270843 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:38.270845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:38.270861 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:38.271186 (+ 325us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:38.271249 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.272253 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.273601 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.274354 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.274433 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:38.274531 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:38.275005 (+ 474us) server_negotiation.cc:299] Negotiation successful
0924 00:24:38.275121 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":118,"threads_started":1}
I20260924 00:24:38.275794 17073 tablet_service.cc:1467] Processing CreateTablet for tablet 95a3ae7518604485b7075ffaf4973012 (DEFAULT_TABLE table=test-table [id=de50eeaa618648e8890d48702773b5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:24:38.275825 16942 tablet_service.cc:1467] Processing CreateTablet for tablet 95a3ae7518604485b7075ffaf4973012 (DEFAULT_TABLE table=test-table [id=de50eeaa618648e8890d48702773b5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:24:38.276253 16942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a3ae7518604485b7075ffaf4973012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:38.276263 17073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a3ae7518604485b7075ffaf4973012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:38.279773 17284 tablet_bootstrap.cc:492] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26: Bootstrap starting.
I20260924 00:24:38.280048 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:38.270842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51498 (local address 127.0.98.3:42813)
0924 00:24:38.271019 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:38.271024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:38.271033 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:24:38.271077 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:38.271080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:38.271098 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:38.271204 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:38.271298 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.272421 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.273830 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.274532 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.274598 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:38.274678 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:38.274744 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 00:24:38.274780 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260924 00:24:38.280036 17285 tablet_bootstrap.cc:492] T 95a3ae7518604485b7075ffaf4973012 P 39a97cd362154ffeb98badca8f1c887f: Bootstrap starting.
I20260924 00:24:38.280563 17284 tablet_bootstrap.cc:654] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:38.280767 17204 tablet_service.cc:1467] Processing CreateTablet for tablet 95a3ae7518604485b7075ffaf4973012 (DEFAULT_TABLE table=test-table [id=de50eeaa618648e8890d48702773b5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:24:38.280889 17284 log.cc:824] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:38.280946 17285 tablet_bootstrap.cc:654] T 95a3ae7518604485b7075ffaf4973012 P 39a97cd362154ffeb98badca8f1c887f: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:38.281118 17204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a3ae7518604485b7075ffaf4973012. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:38.281247 17285 log.cc:824] T 95a3ae7518604485b7075ffaf4973012 P 39a97cd362154ffeb98badca8f1c887f: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:38.281646 17284 tablet_bootstrap.cc:492] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26: No bootstrap required, opened a new log
I20260924 00:24:38.281734 17284 ts_tablet_manager.cc:1397] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:24:38.282157 17285 tablet_bootstrap.cc:492] T 95a3ae7518604485b7075ffaf4973012 P 39a97cd362154ffeb98badca8f1c887f: No bootstrap required, opened a new log
I20260924 00:24:38.282240 17285 ts_tablet_manager.cc:1397] T 95a3ae7518604485b7075ffaf4973012 P 39a97cd362154ffeb98badca8f1c887f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:24:38.284003 17288 tablet_bootstrap.cc:492] T 95a3ae7518604485b7075ffaf4973012 P 6234c49eafc446b8953f7813a2d4e195: Bootstrap starting.
I20260924 00:24:38.284543 17284 raft_consensus.cc:348] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a97cd362154ffeb98badca8f1c887f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46043 } } peers { permanent_uuid: "6234c49eafc446b8953f7813a2d4e195" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "f6d2e5e314aa48868c4735f12a8b4f26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38157 } }
I20260924 00:24:38.284719 17284 raft_consensus.cc:374] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:38.284757 17284 raft_consensus.cc:729] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d2e5e314aa48868c4735f12a8b4f26, State: Initialized, Role: FOLLOWER
I20260924 00:24:38.284797 17288 tablet_bootstrap.cc:654] T 95a3ae7518604485b7075ffaf4973012 P 6234c49eafc446b8953f7813a2d4e195: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:38.284812 17285 raft_consensus.cc:348] T 95a3ae7518604485b7075ffaf4973012 P 39a97cd362154ffeb98badca8f1c887f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a97cd362154ffeb98badca8f1c887f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46043 } } peers { permanent_uuid: "6234c49eafc446b8953f7813a2d4e195" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "f6d2e5e314aa48868c4735f12a8b4f26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38157 } }
I20260924 00:24:38.284906 17284 consensus_queue.cc:260] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a97cd362154ffeb98badca8f1c887f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46043 } } peers { permanent_uuid: "6234c49eafc446b8953f7813a2d4e195" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "f6d2e5e314aa48868c4735f12a8b4f26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38157 } }
I20260924 00:24:38.284958 17285 raft_consensus.cc:374] T 95a3ae7518604485b7075ffaf4973012 P 39a97cd362154ffeb98badca8f1c887f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:38.284991 17285 raft_consensus.cc:729] T 95a3ae7518604485b7075ffaf4973012 P 39a97cd362154ffeb98badca8f1c887f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a97cd362154ffeb98badca8f1c887f, State: Initialized, Role: FOLLOWER
I20260924 00:24:38.285084 17288 log.cc:824] T 95a3ae7518604485b7075ffaf4973012 P 6234c49eafc446b8953f7813a2d4e195: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:38.285167 17284 ts_tablet_manager.cc:1428] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 00:24:38.285127 17285 consensus_queue.cc:260] T 95a3ae7518604485b7075ffaf4973012 P 39a97cd362154ffeb98badca8f1c887f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a97cd362154ffeb98badca8f1c887f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46043 } } peers { permanent_uuid: "6234c49eafc446b8953f7813a2d4e195" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "f6d2e5e314aa48868c4735f12a8b4f26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38157 } }
I20260924 00:24:38.285316 17139 heartbeater.cc:503] Master 127.0.98.62:43103 was elected leader, sending a full tablet report...
I20260924 00:24:38.285395 17285 ts_tablet_manager.cc:1428] T 95a3ae7518604485b7075ffaf4973012 P 39a97cd362154ffeb98badca8f1c887f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:24:38.285506 17008 heartbeater.cc:503] Master 127.0.98.62:43103 was elected leader, sending a full tablet report...
I20260924 00:24:38.286111 17288 tablet_bootstrap.cc:492] T 95a3ae7518604485b7075ffaf4973012 P 6234c49eafc446b8953f7813a2d4e195: No bootstrap required, opened a new log
I20260924 00:24:38.286182 17288 ts_tablet_manager.cc:1397] T 95a3ae7518604485b7075ffaf4973012 P 6234c49eafc446b8953f7813a2d4e195: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:24:38.288448 17288 raft_consensus.cc:348] T 95a3ae7518604485b7075ffaf4973012 P 6234c49eafc446b8953f7813a2d4e195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a97cd362154ffeb98badca8f1c887f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46043 } } peers { permanent_uuid: "6234c49eafc446b8953f7813a2d4e195" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "f6d2e5e314aa48868c4735f12a8b4f26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38157 } }
I20260924 00:24:38.288578 17288 raft_consensus.cc:374] T 95a3ae7518604485b7075ffaf4973012 P 6234c49eafc446b8953f7813a2d4e195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:38.288599 17288 raft_consensus.cc:729] T 95a3ae7518604485b7075ffaf4973012 P 6234c49eafc446b8953f7813a2d4e195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6234c49eafc446b8953f7813a2d4e195, State: Initialized, Role: FOLLOWER
I20260924 00:24:38.288705 17288 consensus_queue.cc:260] T 95a3ae7518604485b7075ffaf4973012 P 6234c49eafc446b8953f7813a2d4e195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a97cd362154ffeb98badca8f1c887f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46043 } } peers { permanent_uuid: "6234c49eafc446b8953f7813a2d4e195" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "f6d2e5e314aa48868c4735f12a8b4f26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38157 } }
I20260924 00:24:38.288959 17288 ts_tablet_manager.cc:1428] T 95a3ae7518604485b7075ffaf4973012 P 6234c49eafc446b8953f7813a2d4e195: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:24:38.289021 17270 heartbeater.cc:503] Master 127.0.98.62:43103 was elected leader, sending a full tablet report...
I20260924 00:24:38.295236 17289 raft_consensus.cc:482] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:24:38.295344 17289 raft_consensus.cc:504] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a97cd362154ffeb98badca8f1c887f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46043 } } peers { permanent_uuid: "6234c49eafc446b8953f7813a2d4e195" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "f6d2e5e314aa48868c4735f12a8b4f26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38157 } }
I20260924 00:24:38.295805 17289 leader_election.cc:288] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39a97cd362154ffeb98badca8f1c887f (127.0.98.1:46043), 6234c49eafc446b8953f7813a2d4e195 (127.0.98.3:42813)
I20260924 00:24:38.299232 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:38.295977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46043 (local address 127.0.98.2:53431)
0924 00:24:38.296034 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:38.296047 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:38.296106 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:38.296306 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:38.296307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:38.296308 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:38.296386 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:38.296388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.297101 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.297102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:38.298231 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:38.298233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.298942 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.298943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:38.299013 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:38.299018 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:24:38.299070 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 00:24:38.299116 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 00:24:38.299255 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:38.296051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53431 (local address 127.0.98.1:46043)
0924 00:24:38.296099 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:38.296101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:38.296111 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:24:38.296205 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:38.296206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:38.296220 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:38.296302 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:38.296477 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.297017 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.298317 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.298859 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.298928 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:38.298937 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:38.299088 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 00:24:38.299132 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:24:38.299383 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:38.296084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42813 (local address 127.0.98.2:56219)
0924 00:24:38.296275 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:38.296284 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:38.296337 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:38.296475 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:38.296475 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:38.296477 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:38.296540 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:38.296542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.297352 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.297353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:38.298486 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:38.298489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.299094 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.299095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:38.299172 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:38.299175 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:24:38.299240 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 00:24:38.299286 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":127,"threads_started":1}
I20260924 00:24:38.299388 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:38.296195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56219 (local address 127.0.98.3:42813)
0924 00:24:38.296248 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:38.296250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:38.296323 (+ 73us) server_negotiation.cc:407] Connection header received
0924 00:24:38.296399 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:38.296400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:38.296410 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:38.296463 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:38.296609 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.297180 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.298578 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.298988 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.299064 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:38.299074 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:38.299254 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 00:24:38.299300 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 00:24:38.299566 16962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a3ae7518604485b7075ffaf4973012" candidate_uuid: "f6d2e5e314aa48868c4735f12a8b4f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a97cd362154ffeb98badca8f1c887f" is_pre_election: true
I20260924 00:24:38.299652 17224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a3ae7518604485b7075ffaf4973012" candidate_uuid: "f6d2e5e314aa48868c4735f12a8b4f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6234c49eafc446b8953f7813a2d4e195" is_pre_election: true
I20260924 00:24:38.299747 16962 raft_consensus.cc:2454] T 95a3ae7518604485b7075ffaf4973012 P 39a97cd362154ffeb98badca8f1c887f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d2e5e314aa48868c4735f12a8b4f26 in term 0.
I20260924 00:24:38.299798 17224 raft_consensus.cc:2454] T 95a3ae7518604485b7075ffaf4973012 P 6234c49eafc446b8953f7813a2d4e195 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d2e5e314aa48868c4735f12a8b4f26 in term 0.
I20260924 00:24:38.300024 17029 leader_election.cc:302] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6234c49eafc446b8953f7813a2d4e195, f6d2e5e314aa48868c4735f12a8b4f26; no voters:
I20260924 00:24:38.300160 17289 raft_consensus.cc:2781] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:24:38.300261 17289 raft_consensus.cc:482] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:24:38.300289 17289 raft_consensus.cc:3037] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:38.301004 17289 raft_consensus.cc:504] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a97cd362154ffeb98badca8f1c887f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46043 } } peers { permanent_uuid: "6234c49eafc446b8953f7813a2d4e195" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "f6d2e5e314aa48868c4735f12a8b4f26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38157 } }
I20260924 00:24:38.301146 17289 leader_election.cc:288] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26 [CANDIDATE]: Term 1 election: Requested vote from peers 39a97cd362154ffeb98badca8f1c887f (127.0.98.1:46043), 6234c49eafc446b8953f7813a2d4e195 (127.0.98.3:42813)
I20260924 00:24:38.301353 16962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a3ae7518604485b7075ffaf4973012" candidate_uuid: "f6d2e5e314aa48868c4735f12a8b4f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a97cd362154ffeb98badca8f1c887f"
I20260924 00:24:38.301354 17224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a3ae7518604485b7075ffaf4973012" candidate_uuid: "f6d2e5e314aa48868c4735f12a8b4f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6234c49eafc446b8953f7813a2d4e195"
I20260924 00:24:38.301437 16962 raft_consensus.cc:3037] T 95a3ae7518604485b7075ffaf4973012 P 39a97cd362154ffeb98badca8f1c887f [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:38.301458 17224 raft_consensus.cc:3037] T 95a3ae7518604485b7075ffaf4973012 P 6234c49eafc446b8953f7813a2d4e195 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:38.302166 17224 raft_consensus.cc:2454] T 95a3ae7518604485b7075ffaf4973012 P 6234c49eafc446b8953f7813a2d4e195 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d2e5e314aa48868c4735f12a8b4f26 in term 1.
I20260924 00:24:38.302295 16962 raft_consensus.cc:2454] T 95a3ae7518604485b7075ffaf4973012 P 39a97cd362154ffeb98badca8f1c887f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d2e5e314aa48868c4735f12a8b4f26 in term 1.
I20260924 00:24:38.302333 17029 leader_election.cc:302] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6234c49eafc446b8953f7813a2d4e195, f6d2e5e314aa48868c4735f12a8b4f26; no voters:
I20260924 00:24:38.302436 17289 raft_consensus.cc:2781] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:38.302600 17289 raft_consensus.cc:686] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26 [term 1 LEADER]: Becoming Leader. State: Replica: f6d2e5e314aa48868c4735f12a8b4f26, State: Running, Role: LEADER
I20260924 00:24:38.302707 17289 consensus_queue.cc:237] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a97cd362154ffeb98badca8f1c887f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46043 } } peers { permanent_uuid: "6234c49eafc446b8953f7813a2d4e195" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } } peers { permanent_uuid: "f6d2e5e314aa48868c4735f12a8b4f26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38157 } }
I20260924 00:24:38.303558 16820 catalog_manager.cc:5495] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6d2e5e314aa48868c4735f12a8b4f26 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f6d2e5e314aa48868c4735f12a8b4f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39a97cd362154ffeb98badca8f1c887f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6234c49eafc446b8953f7813a2d4e195" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d2e5e314aa48868c4735f12a8b4f26" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38157 } health_report { overall_health: HEALTHY } } }
I20260924 00:24:38.316491 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:38.314253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45752 (local address 127.0.98.2:38157)
0924 00:24:38.314318 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:38.314321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:38.314335 (+ 14us) server_negotiation.cc:407] Connection header received
0924 00:24:38.314367 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:38.314368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:38.314388 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:38.314443 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:24:38.314532 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.315000 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.315718 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:38.315915 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:38.315969 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:38.316087 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:24:38.316239 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:24:38.316307 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:38.316394 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 00:24:38.316413 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 00:24:38.318346 16962 raft_consensus.cc:1261] T 95a3ae7518604485b7075ffaf4973012 P 39a97cd362154ffeb98badca8f1c887f [term 1 FOLLOWER]: Refusing update from remote peer f6d2e5e314aa48868c4735f12a8b4f26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:24:38.318372 17224 raft_consensus.cc:1261] T 95a3ae7518604485b7075ffaf4973012 P 6234c49eafc446b8953f7813a2d4e195 [term 1 FOLLOWER]: Refusing update from remote peer f6d2e5e314aa48868c4735f12a8b4f26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:24:38.318758 17289 consensus_queue.cc:1035] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6234c49eafc446b8953f7813a2d4e195" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:24:38.318884 17294 consensus_queue.cc:1035] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39a97cd362154ffeb98badca8f1c887f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:24:38.321452 17298 mvcc.cc:205] Tried to move back new op lower bound from 7332698023189254144 to 7332698023128580096. Current Snapshot: MvccSnapshot[applied={T|T < 7332698023189254144}]
I20260924 00:24:38.323910 16820 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:24:38.324018 16820 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:24:38.324858 16820 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d37336847b74d99b5c028076c2d7c7d: Sending DeleteTablet for 3 replicas of tablet 95a3ae7518604485b7075ffaf4973012
I20260924 00:24:38.325191 16942 tablet_service.cc:1514] Processing DeleteTablet for tablet 95a3ae7518604485b7075ffaf4973012 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:24:38 UTC) from {username='slave'} at 127.0.0.1:60908
I20260924 00:24:38.325263 17204 tablet_service.cc:1514] Processing DeleteTablet for tablet 95a3ae7518604485b7075ffaf4973012 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:24:38 UTC) from {username='slave'} at 127.0.0.1:51498
I20260924 00:24:38.325268 17073 tablet_service.cc:1514] Processing DeleteTablet for tablet 95a3ae7518604485b7075ffaf4973012 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:24:38 UTC) from {username='slave'} at 127.0.0.1:45738
I20260924 00:24:38.325660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 16880
I20260924 00:24:38.330757 17308 tablet_replica.cc:330] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26: stopping tablet replica
I20260924 00:24:38.330880 17308 raft_consensus.cc:2229] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:24:38.331019 17308 raft_consensus.cc:2258] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:24:38.331190 17309 tablet_replica.cc:330] T 95a3ae7518604485b7075ffaf4973012 P 6234c49eafc446b8953f7813a2d4e195: stopping tablet replica
I20260924 00:24:38.331298 17309 raft_consensus.cc:2229] T 95a3ae7518604485b7075ffaf4973012 P 6234c49eafc446b8953f7813a2d4e195 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:24:38.331408 17309 raft_consensus.cc:2258] T 95a3ae7518604485b7075ffaf4973012 P 6234c49eafc446b8953f7813a2d4e195 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:24:38.331705 17308 ts_tablet_manager.cc:1905] T 95a3ae7518604485b7075ffaf4973012 P f6d2e5e314aa48868c4735f12a8b4f26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:24:38.332101 17309 ts_tablet_manager.cc:1905] T 95a3ae7518604485b7075ffaf4973012 P 6234c49eafc446b8953f7813a2d4e195: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 00:24:38.332288 16807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46043 (error 108)
I20260924 00:24:38.332562 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 17011
I20260924 00:24:38.337080 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:38.336906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38157 (local address 127.0.0.1:45754)
0924 00:24:38.336962 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:38.337006 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 00:24:38.337080 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:38.336714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46043 (local address 127.0.0.1:60922)
0924 00:24:38.336962 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:38.337005 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224}
W20260924 00:24:38.337216 16807 catalog_manager.cc:4584] TS f6d2e5e314aa48868c4735f12a8b4f26 (127.0.98.2:38157): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 95a3ae7518604485b7075ffaf4973012: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38157: connect: Connection refused (error 111)
I20260924 00:24:38.337376 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 17142
I20260924 00:24:38.343360 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 16789
I20260924 00:24:38.343546 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:38.343369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42813 (local address 127.0.0.1:51514)
0924 00:24:38.343465 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:38.343502 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:42813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
2026-09-24T00:24:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (686 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:24:38 dist-test-slave-777x krb5kdc[17313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:24:38 dist-test-slave-777x krb5kdc[17313](info): set up 2 sockets
Sep 24 00:24:38 dist-test-slave-777x krb5kdc[17313](info): commencing operation
krb5kdc: starting...
W20260924 00:24:39.471971 392 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:24:39 dist-test-slave-777x krb5kdc[17313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:24:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:24:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:24:39.601114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35263
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:39.683802 17326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:39.684037 17326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:39.684062 17326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:39.685356 17326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:24:39.685410 17326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:39.685425 17326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:39.685438 17326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:24:39.685451 17326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:24:39.687081 17326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:39.687372 17326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:39.687650 17326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:39.693332 17334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:39.693312 17331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:39.693437 17326 server_base.cc:1034] running on GCE node
W20260924 00:24:39.693312 17332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:39.693989 17326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:39.694327 17326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:39.695503 17326 hybrid_clock.cc:648] HybridClock initialized: now 1790209479695503 us; error 43 us; skew 500 ppm
Sep 24 00:24:39 dist-test-slave-777x krb5kdc[17313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:39.698211 17326 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:24:39.698666 17326 webserver.cc:460] Webserver started at http://127.0.98.62:39681/ using document root <none> and password file <none>
I20260924 00:24:39.698956 17326 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:39.699030 17326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:39.699182 17326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:39.700369 17326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "ad118d6f7a72487dab786802f65846b7"
format_stamp: "Formatted at 2026-09-24 00:24:39 on dist-test-slave-777x"
server_key: "96c37f915ef77b3e28a34b0472884fcf"
server_key_iv: "65a2161825827c9054f928e485b5ca7a"
server_key_version: "encryptionkey@0"
I20260924 00:24:39.700735 17326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "ad118d6f7a72487dab786802f65846b7"
format_stamp: "Formatted at 2026-09-24 00:24:39 on dist-test-slave-777x"
server_key: "96c37f915ef77b3e28a34b0472884fcf"
server_key_iv: "65a2161825827c9054f928e485b5ca7a"
server_key_version: "encryptionkey@0"
I20260924 00:24:39.702164 17326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:24:39.702958 17341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:39.703126 17326 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:39.703234 17326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "ad118d6f7a72487dab786802f65846b7"
format_stamp: "Formatted at 2026-09-24 00:24:39 on dist-test-slave-777x"
server_key: "96c37f915ef77b3e28a34b0472884fcf"
server_key_iv: "65a2161825827c9054f928e485b5ca7a"
server_key_version: "encryptionkey@0"
I20260924 00:24:39.703310 17326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:39.723142 17326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:39.725267 17326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:39.725446 17326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:39.729768 17326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35263
I20260924 00:24:39.729800 17393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35263 every 8 connection(s)
I20260924 00:24:39.730201 17326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:24:39.731113 17394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:39.733572 17394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7: Bootstrap starting.
I20260924 00:24:39.734171 17394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:39.734464 17394 log.cc:824] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:39.735204 17394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7: No bootstrap required, opened a new log
I20260924 00:24:39.736724 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 17326
I20260924 00:24:39.736821 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:24:39.737056 392 external_mini_cluster.cc:1428] Setting key bce955bb74dd51140289612e58a265e5
I20260924 00:24:39.737063 17394 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad118d6f7a72487dab786802f65846b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35263 } }
I20260924 00:24:39.737259 17394 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:39.737308 17394 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad118d6f7a72487dab786802f65846b7, State: Initialized, Role: FOLLOWER
I20260924 00:24:39.737483 17394 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad118d6f7a72487dab786802f65846b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35263 } }
I20260924 00:24:39.737564 17394 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:24:39.737596 17394 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:24:39.737638 17394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:39.738267 17394 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad118d6f7a72487dab786802f65846b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35263 } }
I20260924 00:24:39.738360 17394 leader_election.cc:302] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad118d6f7a72487dab786802f65846b7; no voters:
I20260924 00:24:39.738554 17394 leader_election.cc:288] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:24:39.738582 17399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:39.738826 17399 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7 [term 1 LEADER]: Becoming Leader. State: Replica: ad118d6f7a72487dab786802f65846b7, State: Running, Role: LEADER
I20260924 00:24:39.738843 17394 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:24:39.739007 17399 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad118d6f7a72487dab786802f65846b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35263 } }
I20260924 00:24:39.740507 17401 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad118d6f7a72487dab786802f65846b7. Latest consensus state: current_term: 1 leader_uuid: "ad118d6f7a72487dab786802f65846b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad118d6f7a72487dab786802f65846b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35263 } } }
I20260924 00:24:39.740520 17400 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad118d6f7a72487dab786802f65846b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad118d6f7a72487dab786802f65846b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35263 } } }
I20260924 00:24:39.740728 17400 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7 [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:39.740657 17401 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7 [sys.catalog]: This master's current role is: LEADER
W20260924 00:24:39.740867 17414 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:24:39.740913 17414 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:24:39.740952 17416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:24:39.741154 17416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:24:39.743829 17416 catalog_manager.cc:1347] Generated new cluster ID: e0f059e9eb76456fa8ba8963fa7b76d1
I20260924 00:24:39.743896 17416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:24:39 dist-test-slave-777x krb5kdc[17313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:24:39.750337 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:39.737652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46674 (local address 127.0.98.62:35263)
0924 00:24:39.737850 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:39.737861 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:39.737897 (+ 36us) server_negotiation.cc:407] Connection header received
0924 00:24:39.738262 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:39.738269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:39.738318 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:39.738521 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:39.739369 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:39.740929 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:39.741886 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:39.742542 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:39.742663 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:39.745646 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:39.745651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:39.745651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:39.745665 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:39.748300 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:39.748839 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:39.748843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:39.748843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:39.748983 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:39.749206 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:39.749207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:39.749207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:39.749532 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:39.749693 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:39.749823 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 00:24:39.750089 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":112,"threads_started":1}
I20260924 00:24:39.754988 17416 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:24:39.755738 17416 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:24:39.765038 17416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7: Generated new TSK 0
I20260924 00:24:39.765285 17416 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:24:39.799454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35263
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:39.882390 17420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:39.882642 17420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:39.882684 17420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:39.883880 17420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:39.883951 17420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:39.883996 17420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:24:39.885537 17420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:39.885823 17420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:39.886057 17420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:39.891319 17426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:39.891374 17425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:39.891336 17420 server_base.cc:1034] running on GCE node
W20260924 00:24:39.891309 17428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:39.891830 17420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:39.892055 17420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:39.893225 17420 hybrid_clock.cc:648] HybridClock initialized: now 1790209479893184 us; error 37 us; skew 500 ppm
Sep 24 00:24:39 dist-test-slave-777x krb5kdc[17313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:39.895879 17420 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:24:39.896461 17420 webserver.cc:460] Webserver started at http://127.0.98.1:40803/ using document root <none> and password file <none>
I20260924 00:24:39.896705 17420 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:39.896751 17420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:39.896904 17420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:39.898010 17420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "a2fb45e108364c848f970a814430a48b"
format_stamp: "Formatted at 2026-09-24 00:24:39 on dist-test-slave-777x"
server_key: "d46ca2118c29c0f3f9711bcd34e6d8e2"
server_key_iv: "2f3f6cc5174b5109510cb83352e8fa4c"
server_key_version: "encryptionkey@0"
I20260924 00:24:39.898386 17420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a2fb45e108364c848f970a814430a48b"
format_stamp: "Formatted at 2026-09-24 00:24:39 on dist-test-slave-777x"
server_key: "d46ca2118c29c0f3f9711bcd34e6d8e2"
server_key_iv: "2f3f6cc5174b5109510cb83352e8fa4c"
server_key_version: "encryptionkey@0"
I20260924 00:24:39.899740 17420 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:24:39.900594 17435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:39.900761 17420 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:24:39.900868 17420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "a2fb45e108364c848f970a814430a48b"
format_stamp: "Formatted at 2026-09-24 00:24:39 on dist-test-slave-777x"
server_key: "d46ca2118c29c0f3f9711bcd34e6d8e2"
server_key_iv: "2f3f6cc5174b5109510cb83352e8fa4c"
server_key_version: "encryptionkey@0"
I20260924 00:24:39.900944 17420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:39.918587 17420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:39.920646 17420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:39.920823 17420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:39.921072 17420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:24:39.921404 17420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:24:39.921458 17420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:39.921494 17420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:24:39.921519 17420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:39.927673 17420 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35319
I20260924 00:24:39.927724 17548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35319 every 8 connection(s)
I20260924 00:24:39.928041 17420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 00:24:39 dist-test-slave-777x krb5kdc[17313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:24:39.934926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 17420
I20260924 00:24:39.935034 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:24:39.935266 392 external_mini_cluster.cc:1428] Setting key fe46883ba603ead9d35b31e71eccf2c8
I20260924 00:24:39.938107 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:39.928868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32777 (local address 127.0.98.62:35263)
0924 00:24:39.928964 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:39.928966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:39.929125 (+ 159us) server_negotiation.cc:407] Connection header received
0924 00:24:39.930135 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:39.930136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:39.930150 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:39.930211 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:39.930861 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:39.931389 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:39.932447 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:39.932667 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:39.932757 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:39.934928 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:39.934930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:39.934930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:39.934937 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:39.936537 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:39.937023 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:39.937024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:39.937024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:39.937074 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:39.937425 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:39.937427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:39.937427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:39.937558 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:39.937640 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:39.937930 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 00:24:39.938018 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 00:24:39.938308 17551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:39.928790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35263 (local address 127.0.98.1:32777)
0924 00:24:39.928982 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:39.929028 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:39.930013 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:39.930284 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:39.930295 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:39.930563 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:24:39.930757 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:39.930761 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:39.931493 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:39.931494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:39.932353 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:39.932355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:39.932770 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:39.932771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:39.932911 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:39.933421 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:24:39.933430 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:24:39.934822 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:24:39.936612 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:24:39.936619 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:24:39.936622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:24:39.936945 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:24:39.937141 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:24:39.937141 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:24:39.937142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:24:39.937319 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:24:39.937631 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:24:39.937637 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:24:39.937785 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 00:24:39.937896 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 00:24:39.938057 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20260924 00:24:39.938843 17549 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35263
I20260924 00:24:39.938992 17549 heartbeater.cc:464] Registering TS with master...
I20260924 00:24:39.939177 17549 heartbeater.cc:511] Master 127.0.98.62:35263 requested a full tablet report, sending...
I20260924 00:24:39.939687 17358 ts_manager.cc:194] Registered new tserver with Master: a2fb45e108364c848f970a814430a48b (127.0.98.1:35319)
I20260924 00:24:39.940451 17358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32777
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:24:39.963413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35263
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:40.046144 17554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:40.046398 17554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:40.046437 17554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:40.047662 17554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:40.047870 17554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:40.047920 17554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:24:40.049522 17554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:40.049810 17554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:40.050055 17554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:40.055234 17560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:40.055413 17559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:40.055516 17554 server_base.cc:1034] running on GCE node
W20260924 00:24:40.055413 17562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:40.055819 17554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:40.056093 17554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:40.057264 17554 hybrid_clock.cc:648] HybridClock initialized: now 1790209480057255 us; error 65 us; skew 500 ppm
Sep 24 00:24:40 dist-test-slave-777x krb5kdc[17313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:40.059912 17554 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:24:40.060441 17554 webserver.cc:460] Webserver started at http://127.0.98.2:41789/ using document root <none> and password file <none>
I20260924 00:24:40.060731 17554 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:40.060809 17554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:40.060963 17554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:40.062019 17554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "dd7fac8544d64a7dacb21fc9ee57dc56"
format_stamp: "Formatted at 2026-09-24 00:24:40 on dist-test-slave-777x"
server_key: "788742f5a7e2f387a9a1691d5843d262"
server_key_iv: "bb9eaeb3c4b9231fb8129187098057a2"
server_key_version: "encryptionkey@0"
I20260924 00:24:40.062399 17554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "dd7fac8544d64a7dacb21fc9ee57dc56"
format_stamp: "Formatted at 2026-09-24 00:24:40 on dist-test-slave-777x"
server_key: "788742f5a7e2f387a9a1691d5843d262"
server_key_iv: "bb9eaeb3c4b9231fb8129187098057a2"
server_key_version: "encryptionkey@0"
I20260924 00:24:40.063905 17554 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:24:40.064870 17569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:40.065086 17554 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:24:40.065189 17554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "dd7fac8544d64a7dacb21fc9ee57dc56"
format_stamp: "Formatted at 2026-09-24 00:24:40 on dist-test-slave-777x"
server_key: "788742f5a7e2f387a9a1691d5843d262"
server_key_iv: "bb9eaeb3c4b9231fb8129187098057a2"
server_key_version: "encryptionkey@0"
I20260924 00:24:40.065280 17554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:40.077190 17554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:40.079587 17554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:40.079761 17554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:40.079986 17554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:24:40.080348 17554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:24:40.080401 17554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:40.080444 17554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:24:40.080470 17554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:40.086479 17554 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39223
I20260924 00:24:40.086522 17682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39223 every 8 connection(s)
I20260924 00:24:40.086880 17554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:24:40.088007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 17554
I20260924 00:24:40.088124 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:24:40.088351 392 external_mini_cluster.cc:1428] Setting key 52ad68df8dc8d9ad838b43377269f848
Sep 24 00:24:40 dist-test-slave-777x krb5kdc[17313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 00:24:40.098271 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:40.087692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57907 (local address 127.0.98.62:35263)
0924 00:24:40.087756 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:40.087758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:40.087956 (+ 198us) server_negotiation.cc:407] Connection header received
0924 00:24:40.089098 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:40.089099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:40.089112 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:40.089166 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:40.089926 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.090901 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.092202 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.092552 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.092633 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:40.095154 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:40.095156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:40.095156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:40.095166 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:40.096841 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:40.097300 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:40.097301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:40.097302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:40.097364 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:40.097593 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:40.097594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:40.097594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:40.097695 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:40.097741 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:40.098045 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 00:24:40.098191 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 00:24:40.098455 17685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:40.087648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35263 (local address 127.0.98.2:57907)
0924 00:24:40.087812 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:40.087853 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:40.088972 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:40.089236 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:40.089240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:40.089517 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:24:40.089810 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:40.089817 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.091002 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.091003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:40.092084 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:40.092087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.092652 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.092654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:40.092796 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:40.093386 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:24:40.093395 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:24:40.095047 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:24:40.096938 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:24:40.096944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:24:40.096946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:24:40.097218 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:24:40.097443 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:24:40.097444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:24:40.097444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:24:40.097550 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:24:40.097746 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:24:40.097750 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:24:40.097896 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 00:24:40.097984 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 00:24:40.098109 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260924 00:24:40.099035 17683 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35263
I20260924 00:24:40.099135 17683 heartbeater.cc:464] Registering TS with master...
I20260924 00:24:40.099318 17683 heartbeater.cc:511] Master 127.0.98.62:35263 requested a full tablet report, sending...
I20260924 00:24:40.099803 17358 ts_manager.cc:194] Registered new tserver with Master: dd7fac8544d64a7dacb21fc9ee57dc56 (127.0.98.2:39223)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 00:24:40.100312 17358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:24:40.118574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35263
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:40.200747 17688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:40.200968 17688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:40.200992 17688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:40.202189 17688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:40.202239 17688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:40.202266 17688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:24:40.204066 17688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:40.204356 17688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:40.204615 17688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:40.210254 17696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:40.210253 17693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:40.210290 17688 server_base.cc:1034] running on GCE node
W20260924 00:24:40.210247 17694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:40.210701 17688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:40.210973 17688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:40.212150 17688 hybrid_clock.cc:648] HybridClock initialized: now 1790209480212136 us; error 47 us; skew 500 ppm
Sep 24 00:24:40 dist-test-slave-777x krb5kdc[17313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:40.214748 17688 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:24:40.215214 17688 webserver.cc:460] Webserver started at http://127.0.98.3:41571/ using document root <none> and password file <none>
I20260924 00:24:40.215456 17688 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:40.215503 17688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:40.215664 17688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:40.216856 17688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "687cca53581f4b639681c9b19ef59627"
format_stamp: "Formatted at 2026-09-24 00:24:40 on dist-test-slave-777x"
server_key: "5c398412bf05c7c83741ae0742a09fe7"
server_key_iv: "78a8817a3ef800da7804f18176a20938"
server_key_version: "encryptionkey@0"
I20260924 00:24:40.217221 17688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "687cca53581f4b639681c9b19ef59627"
format_stamp: "Formatted at 2026-09-24 00:24:40 on dist-test-slave-777x"
server_key: "5c398412bf05c7c83741ae0742a09fe7"
server_key_iv: "78a8817a3ef800da7804f18176a20938"
server_key_version: "encryptionkey@0"
I20260924 00:24:40.218580 17688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:24:40.219319 17703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:40.219496 17688 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:24:40.219576 17688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "687cca53581f4b639681c9b19ef59627"
format_stamp: "Formatted at 2026-09-24 00:24:40 on dist-test-slave-777x"
server_key: "5c398412bf05c7c83741ae0742a09fe7"
server_key_iv: "78a8817a3ef800da7804f18176a20938"
server_key_version: "encryptionkey@0"
I20260924 00:24:40.219630 17688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:40.226852 17688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:40.229166 17688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:40.229278 17688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:40.229461 17688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:24:40.229710 17688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:24:40.229745 17688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:40.229768 17688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:24:40.229779 17688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:40.235523 17688 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39627
I20260924 00:24:40.235607 17816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39627 every 8 connection(s)
I20260924 00:24:40.235926 17688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 00:24:40 dist-test-slave-777x krb5kdc[17313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:24:40.243590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 17688
I20260924 00:24:40.243731 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:24:40.243975 392 external_mini_cluster.cc:1428] Setting key 7613ae38952fede21d6b842d688ab5cd
I20260924 00:24:40.245803 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:40.236703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52437 (local address 127.0.98.62:35263)
0924 00:24:40.236751 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:40.236753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:40.236964 (+ 211us) server_negotiation.cc:407] Connection header received
0924 00:24:40.237864 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:40.237865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:40.237877 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:40.237926 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:40.238497 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.239079 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.240229 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.240454 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.240547 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:40.242775 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:40.242777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:40.242777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:40.242785 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:40.244494 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:40.245051 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:40.245052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:40.245052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:40.245103 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:40.245285 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:40.245286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:40.245286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:40.245390 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:40.245440 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:40.245678 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 00:24:40.245740 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 00:24:40.245993 17819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:40.236658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35263 (local address 127.0.98.3:52437)
0924 00:24:40.236828 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:40.236864 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:40.237765 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:40.237988 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:40.237991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:40.238234 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:24:40.238428 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:40.238432 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.239201 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.239202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:40.240100 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:40.240103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.240551 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.240552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:40.240681 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:40.241277 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:24:40.241286 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:24:40.242668 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:24:40.244653 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:24:40.244660 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:24:40.244663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:24:40.244999 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:24:40.245145 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:24:40.245145 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:24:40.245146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:24:40.245233 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:24:40.245447 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:24:40.245452 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:24:40.245567 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 00:24:40.245650 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 00:24:40.245771 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":63,"threads_started":1}
I20260924 00:24:40.246459 17817 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35263
I20260924 00:24:40.246539 17817 heartbeater.cc:464] Registering TS with master...
I20260924 00:24:40.246678 17817 heartbeater.cc:511] Master 127.0.98.62:35263 requested a full tablet report, sending...
I20260924 00:24:40.246985 17358 ts_manager.cc:194] Registered new tserver with Master: 687cca53581f4b639681c9b19ef59627 (127.0.98.3:39627)
I20260924 00:24:40.247344 17358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52437
I20260924 00:24:40.255945 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:24:40.264192 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:40.257379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46686 (local address 127.0.98.62:35263)
0924 00:24:40.257479 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:40.257481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:40.257589 (+ 108us) server_negotiation.cc:407] Connection header received
0924 00:24:40.257659 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:40.257660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:40.257673 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:40.257740 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:40.258373 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.258864 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.259611 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.259804 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.259893 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:40.261231 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:40.261233 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:40.261233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:40.261241 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:40.262760 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:40.263262 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:40.263263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:40.263263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:40.263328 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:40.263582 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:40.263583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:40.263583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:40.263748 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:40.263874 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:40.263998 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 00:24:40.264064 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 00:24:40.266074 17358 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:24:40.267604 17358 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:24:40.274462 17828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:40.269932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39223 (local address 127.0.0.1:44248)
0924 00:24:40.270549 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:40.270557 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:40.270582 (+ 25us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:40.271197 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:40.271198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:40.271200 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:40.271238 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:40.271239 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.272053 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.272054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:40.273404 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:40.273407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.274129 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.274130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:40.274214 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:40.274229 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 00:24:40.274298 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 00:24:40.274364 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":45,"thread_start_us":84,"threads_started":1}
I20260924 00:24:40.274629 17829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:40.269932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39627 (local address 127.0.0.1:51968)
0924 00:24:40.270092 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:40.270382 (+ 290us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:40.270466 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:40.270690 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:40.270694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:40.270697 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:40.270771 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:40.270773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.272128 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.272130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:40.273224 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:40.273226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.274234 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.274235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:40.274320 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:40.274324 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:24:40.274411 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 00:24:40.274464 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":1,"thread_start_us":44,"threads_started":1}
I20260924 00:24:40.274645 17831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:40.270090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51968 (local address 127.0.98.3:39627)
0924 00:24:40.270279 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:40.270291 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:40.270456 (+ 165us) server_negotiation.cc:407] Connection header received
0924 00:24:40.270536 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:40.270537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:40.270550 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:40.270673 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:40.270833 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.272009 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.273317 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.274069 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.274144 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:40.274204 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:40.274511 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 00:24:40.274565 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":77,"threads_started":1}
I20260924 00:24:40.274784 17830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:40.269972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35319 (local address 127.0.0.1:56800)
0924 00:24:40.270629 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:40.270635 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:40.270695 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:40.270890 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:40.270890 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:40.270893 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:40.270945 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:40.270947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.272263 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.272264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:40.273404 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:40.273407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.274301 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.274302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:40.274470 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:40.274476 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 00:24:40.274538 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 00:24:40.274710 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":5,"thread_start_us":77,"threads_started":1}
I20260924 00:24:40.274878 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:40.270092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44248 (local address 127.0.98.2:39223)
0924 00:24:40.271041 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:40.271046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:40.271055 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:24:40.271092 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:40.271094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:40.271107 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:40.271200 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:40.271284 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.271949 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.273538 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.274027 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.274118 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:40.274187 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:40.274731 (+ 544us) server_negotiation.cc:299] Negotiation successful
0924 00:24:40.274787 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":907,"thread_start_us":78,"threads_started":1}
I20260924 00:24:40.275511 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:40.270201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56800 (local address 127.0.98.1:35319)
0924 00:24:40.270367 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:40.270372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:40.270749 (+ 377us) server_negotiation.cc:407] Connection header received
0924 00:24:40.270797 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:40.270800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:40.270816 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:40.270894 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:40.271098 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.272132 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.273495 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.274214 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.274285 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:40.274342 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:40.275369 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0924 00:24:40.275424 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":116,"threads_started":1}
I20260924 00:24:40.275490 17751 tablet_service.cc:1467] Processing CreateTablet for tablet 94873eb662834e098d787c1eb4015959 (DEFAULT_TABLE table=test-table [id=3cb61fed8180469ebb2f5b0fd39c6c0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:24:40.275487 17617 tablet_service.cc:1467] Processing CreateTablet for tablet 94873eb662834e098d787c1eb4015959 (DEFAULT_TABLE table=test-table [id=3cb61fed8180469ebb2f5b0fd39c6c0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:24:40.275869 17617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94873eb662834e098d787c1eb4015959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:40.275873 17751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94873eb662834e098d787c1eb4015959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:40.276094 17483 tablet_service.cc:1467] Processing CreateTablet for tablet 94873eb662834e098d787c1eb4015959 (DEFAULT_TABLE table=test-table [id=3cb61fed8180469ebb2f5b0fd39c6c0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:24:40.276453 17483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94873eb662834e098d787c1eb4015959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:40.278815 17834 tablet_bootstrap.cc:492] T 94873eb662834e098d787c1eb4015959 P dd7fac8544d64a7dacb21fc9ee57dc56: Bootstrap starting.
I20260924 00:24:40.279486 17834 tablet_bootstrap.cc:654] T 94873eb662834e098d787c1eb4015959 P dd7fac8544d64a7dacb21fc9ee57dc56: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:40.279567 17835 tablet_bootstrap.cc:492] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627: Bootstrap starting.
I20260924 00:24:40.279906 17836 tablet_bootstrap.cc:492] T 94873eb662834e098d787c1eb4015959 P a2fb45e108364c848f970a814430a48b: Bootstrap starting.
I20260924 00:24:40.280269 17834 log.cc:824] T 94873eb662834e098d787c1eb4015959 P dd7fac8544d64a7dacb21fc9ee57dc56: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:40.280316 17835 tablet_bootstrap.cc:654] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:40.280644 17836 tablet_bootstrap.cc:654] T 94873eb662834e098d787c1eb4015959 P a2fb45e108364c848f970a814430a48b: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:40.280823 17835 log.cc:824] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:40.281235 17834 tablet_bootstrap.cc:492] T 94873eb662834e098d787c1eb4015959 P dd7fac8544d64a7dacb21fc9ee57dc56: No bootstrap required, opened a new log
I20260924 00:24:40.281226 17836 log.cc:824] T 94873eb662834e098d787c1eb4015959 P a2fb45e108364c848f970a814430a48b: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:40.281335 17834 ts_tablet_manager.cc:1397] T 94873eb662834e098d787c1eb4015959 P dd7fac8544d64a7dacb21fc9ee57dc56: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 00:24:40.281595 17835 tablet_bootstrap.cc:492] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627: No bootstrap required, opened a new log
I20260924 00:24:40.281673 17835 ts_tablet_manager.cc:1397] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 00:24:40.282264 17836 tablet_bootstrap.cc:492] T 94873eb662834e098d787c1eb4015959 P a2fb45e108364c848f970a814430a48b: No bootstrap required, opened a new log
I20260924 00:24:40.282336 17836 ts_tablet_manager.cc:1397] T 94873eb662834e098d787c1eb4015959 P a2fb45e108364c848f970a814430a48b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:24:40.283002 17834 raft_consensus.cc:348] T 94873eb662834e098d787c1eb4015959 P dd7fac8544d64a7dacb21fc9ee57dc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7fac8544d64a7dacb21fc9ee57dc56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39223 } } peers { permanent_uuid: "687cca53581f4b639681c9b19ef59627" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39627 } } peers { permanent_uuid: "a2fb45e108364c848f970a814430a48b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35319 } }
I20260924 00:24:40.283172 17834 raft_consensus.cc:374] T 94873eb662834e098d787c1eb4015959 P dd7fac8544d64a7dacb21fc9ee57dc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:40.283216 17834 raft_consensus.cc:729] T 94873eb662834e098d787c1eb4015959 P dd7fac8544d64a7dacb21fc9ee57dc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7fac8544d64a7dacb21fc9ee57dc56, State: Initialized, Role: FOLLOWER
I20260924 00:24:40.283337 17834 consensus_queue.cc:260] T 94873eb662834e098d787c1eb4015959 P dd7fac8544d64a7dacb21fc9ee57dc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7fac8544d64a7dacb21fc9ee57dc56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39223 } } peers { permanent_uuid: "687cca53581f4b639681c9b19ef59627" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39627 } } peers { permanent_uuid: "a2fb45e108364c848f970a814430a48b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35319 } }
I20260924 00:24:40.283648 17834 ts_tablet_manager.cc:1428] T 94873eb662834e098d787c1eb4015959 P dd7fac8544d64a7dacb21fc9ee57dc56: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 00:24:40.283648 17683 heartbeater.cc:503] Master 127.0.98.62:35263 was elected leader, sending a full tablet report...
I20260924 00:24:40.283749 17835 raft_consensus.cc:348] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7fac8544d64a7dacb21fc9ee57dc56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39223 } } peers { permanent_uuid: "687cca53581f4b639681c9b19ef59627" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39627 } } peers { permanent_uuid: "a2fb45e108364c848f970a814430a48b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35319 } }
I20260924 00:24:40.283933 17835 raft_consensus.cc:374] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:40.284106 17835 raft_consensus.cc:729] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 687cca53581f4b639681c9b19ef59627, State: Initialized, Role: FOLLOWER
I20260924 00:24:40.284264 17835 consensus_queue.cc:260] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7fac8544d64a7dacb21fc9ee57dc56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39223 } } peers { permanent_uuid: "687cca53581f4b639681c9b19ef59627" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39627 } } peers { permanent_uuid: "a2fb45e108364c848f970a814430a48b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35319 } }
I20260924 00:24:40.284384 17836 raft_consensus.cc:348] T 94873eb662834e098d787c1eb4015959 P a2fb45e108364c848f970a814430a48b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7fac8544d64a7dacb21fc9ee57dc56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39223 } } peers { permanent_uuid: "687cca53581f4b639681c9b19ef59627" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39627 } } peers { permanent_uuid: "a2fb45e108364c848f970a814430a48b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35319 } }
I20260924 00:24:40.284552 17835 ts_tablet_manager.cc:1428] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 00:24:40.284554 17836 raft_consensus.cc:374] T 94873eb662834e098d787c1eb4015959 P a2fb45e108364c848f970a814430a48b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:40.284596 17836 raft_consensus.cc:729] T 94873eb662834e098d787c1eb4015959 P a2fb45e108364c848f970a814430a48b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2fb45e108364c848f970a814430a48b, State: Initialized, Role: FOLLOWER
I20260924 00:24:40.284606 17817 heartbeater.cc:503] Master 127.0.98.62:35263 was elected leader, sending a full tablet report...
I20260924 00:24:40.284708 17836 consensus_queue.cc:260] T 94873eb662834e098d787c1eb4015959 P a2fb45e108364c848f970a814430a48b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7fac8544d64a7dacb21fc9ee57dc56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39223 } } peers { permanent_uuid: "687cca53581f4b639681c9b19ef59627" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39627 } } peers { permanent_uuid: "a2fb45e108364c848f970a814430a48b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35319 } }
I20260924 00:24:40.284993 17836 ts_tablet_manager.cc:1428] T 94873eb662834e098d787c1eb4015959 P a2fb45e108364c848f970a814430a48b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:24:40.285024 17549 heartbeater.cc:503] Master 127.0.98.62:35263 was elected leader, sending a full tablet report...
W20260924 00:24:40.337489 17684 tablet.cc:2369] T 94873eb662834e098d787c1eb4015959 P dd7fac8544d64a7dacb21fc9ee57dc56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:24:40.411861 17841 raft_consensus.cc:482] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:24:40.411993 17841 raft_consensus.cc:504] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7fac8544d64a7dacb21fc9ee57dc56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39223 } } peers { permanent_uuid: "687cca53581f4b639681c9b19ef59627" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39627 } } peers { permanent_uuid: "a2fb45e108364c848f970a814430a48b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35319 } }
I20260924 00:24:40.412405 17841 leader_election.cc:288] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7fac8544d64a7dacb21fc9ee57dc56 (127.0.98.2:39223), a2fb45e108364c848f970a814430a48b (127.0.98.1:35319)
I20260924 00:24:40.416396 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:40.412686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58663 (local address 127.0.98.2:39223)
0924 00:24:40.412736 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:40.412738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:40.412752 (+ 14us) server_negotiation.cc:407] Connection header received
0924 00:24:40.412843 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:40.412844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:40.412854 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:40.412899 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:40.413113 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.413872 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.415409 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.415978 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.416057 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:40.416067 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:40.416271 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 00:24:40.416324 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 00:24:40.416419 17819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:40.412621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39223 (local address 127.0.98.3:58663)
0924 00:24:40.412689 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:40.412703 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:40.412750 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:40.412926 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:40.412928 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:40.412930 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:40.413015 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:40.413017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.413981 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.413983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:40.415303 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:40.415305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.416067 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.416068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:40.416142 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:40.416145 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:24:40.416262 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 00:24:40.416331 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":32}
I20260924 00:24:40.416464 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:40.412704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37167 (local address 127.0.98.1:35319)
0924 00:24:40.412745 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:40.412747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:40.412946 (+ 199us) server_negotiation.cc:407] Connection header received
0924 00:24:40.412983 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:40.412984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:40.412994 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:40.413037 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:40.413187 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.413928 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.415409 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.415986 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.416057 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:40.416067 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:40.416349 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 00:24:40.416390 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:24:40.416752 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:40.412621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35319 (local address 127.0.98.3:37167)
0924 00:24:40.412855 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:40.412863 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:40.412932 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:40.413061 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:40.413062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:40.413064 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:40.413122 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:40.413123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.414007 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.414008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:40.415298 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:40.415302 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.416130 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.416131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:40.416264 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:40.416269 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:24:40.416328 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 00:24:40.416367 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":70,"thread_start_us":110,"threads_started":1}
I20260924 00:24:40.417008 17637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94873eb662834e098d787c1eb4015959" candidate_uuid: "687cca53581f4b639681c9b19ef59627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7fac8544d64a7dacb21fc9ee57dc56" is_pre_election: true
I20260924 00:24:40.417060 17503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94873eb662834e098d787c1eb4015959" candidate_uuid: "687cca53581f4b639681c9b19ef59627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2fb45e108364c848f970a814430a48b" is_pre_election: true
I20260924 00:24:40.417179 17637 raft_consensus.cc:2454] T 94873eb662834e098d787c1eb4015959 P dd7fac8544d64a7dacb21fc9ee57dc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 687cca53581f4b639681c9b19ef59627 in term 0.
I20260924 00:24:40.417228 17503 raft_consensus.cc:2454] T 94873eb662834e098d787c1eb4015959 P a2fb45e108364c848f970a814430a48b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 687cca53581f4b639681c9b19ef59627 in term 0.
I20260924 00:24:40.417443 17707 leader_election.cc:302] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 687cca53581f4b639681c9b19ef59627, a2fb45e108364c848f970a814430a48b; no voters:
I20260924 00:24:40.417559 17841 raft_consensus.cc:2781] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:24:40.417610 17841 raft_consensus.cc:482] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:24:40.417634 17841 raft_consensus.cc:3037] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:40.418325 17841 raft_consensus.cc:504] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7fac8544d64a7dacb21fc9ee57dc56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39223 } } peers { permanent_uuid: "687cca53581f4b639681c9b19ef59627" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39627 } } peers { permanent_uuid: "a2fb45e108364c848f970a814430a48b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35319 } }
I20260924 00:24:40.418468 17841 leader_election.cc:288] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627 [CANDIDATE]: Term 1 election: Requested vote from peers dd7fac8544d64a7dacb21fc9ee57dc56 (127.0.98.2:39223), a2fb45e108364c848f970a814430a48b (127.0.98.1:35319)
I20260924 00:24:40.418625 17637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94873eb662834e098d787c1eb4015959" candidate_uuid: "687cca53581f4b639681c9b19ef59627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7fac8544d64a7dacb21fc9ee57dc56"
I20260924 00:24:40.418625 17503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94873eb662834e098d787c1eb4015959" candidate_uuid: "687cca53581f4b639681c9b19ef59627" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2fb45e108364c848f970a814430a48b"
I20260924 00:24:40.418702 17637 raft_consensus.cc:3037] T 94873eb662834e098d787c1eb4015959 P dd7fac8544d64a7dacb21fc9ee57dc56 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:40.418702 17503 raft_consensus.cc:3037] T 94873eb662834e098d787c1eb4015959 P a2fb45e108364c848f970a814430a48b [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:40.419550 17503 raft_consensus.cc:2454] T 94873eb662834e098d787c1eb4015959 P a2fb45e108364c848f970a814430a48b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 687cca53581f4b639681c9b19ef59627 in term 1.
I20260924 00:24:40.419550 17637 raft_consensus.cc:2454] T 94873eb662834e098d787c1eb4015959 P dd7fac8544d64a7dacb21fc9ee57dc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 687cca53581f4b639681c9b19ef59627 in term 1.
I20260924 00:24:40.419724 17707 leader_election.cc:302] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 687cca53581f4b639681c9b19ef59627, a2fb45e108364c848f970a814430a48b; no voters:
I20260924 00:24:40.419862 17841 raft_consensus.cc:2781] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:40.420025 17841 raft_consensus.cc:686] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627 [term 1 LEADER]: Becoming Leader. State: Replica: 687cca53581f4b639681c9b19ef59627, State: Running, Role: LEADER
I20260924 00:24:40.420125 17841 consensus_queue.cc:237] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7fac8544d64a7dacb21fc9ee57dc56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39223 } } peers { permanent_uuid: "687cca53581f4b639681c9b19ef59627" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39627 } } peers { permanent_uuid: "a2fb45e108364c848f970a814430a48b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35319 } }
I20260924 00:24:40.421072 17357 catalog_manager.cc:5495] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627 reported cstate change: term changed from 0 to 1, leader changed from <none> to 687cca53581f4b639681c9b19ef59627 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "687cca53581f4b639681c9b19ef59627" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7fac8544d64a7dacb21fc9ee57dc56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "687cca53581f4b639681c9b19ef59627" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2fb45e108364c848f970a814430a48b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35319 } health_report { overall_health: UNKNOWN } } }
W20260924 00:24:40.428830 17550 tablet.cc:2369] T 94873eb662834e098d787c1eb4015959 P a2fb45e108364c848f970a814430a48b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:24:40.457506 17831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:40.454865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51970 (local address 127.0.98.3:39627)
0924 00:24:40.454935 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:40.454938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:40.454947 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:24:40.454991 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:40.454991 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:40.455013 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:40.455102 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:24:40.455257 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.455765 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.456667 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.456879 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.456953 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:40.457122 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:24:40.457226 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:24:40.457341 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:40.457391 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 00:24:40.457399 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 00:24:40.459343 17503 raft_consensus.cc:1261] T 94873eb662834e098d787c1eb4015959 P a2fb45e108364c848f970a814430a48b [term 1 FOLLOWER]: Refusing update from remote peer 687cca53581f4b639681c9b19ef59627: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:24:40.459376 17637 raft_consensus.cc:1261] T 94873eb662834e098d787c1eb4015959 P dd7fac8544d64a7dacb21fc9ee57dc56 [term 1 FOLLOWER]: Refusing update from remote peer 687cca53581f4b639681c9b19ef59627: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:24:40.459761 17844 consensus_queue.cc:1035] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2fb45e108364c848f970a814430a48b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:24:40.459911 17841 consensus_queue.cc:1035] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7fac8544d64a7dacb21fc9ee57dc56" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:24:40.463016 17849 mvcc.cc:205] Tried to move back new op lower bound from 7332698031958470656 to 7332698031801815040. Current Snapshot: MvccSnapshot[applied={T|T < 7332698031958470656 or (T in {7332698031958470656})}]
I20260924 00:24:40.463064 17848 mvcc.cc:205] Tried to move back new op lower bound from 7332698031958470656 to 7332698031801815040. Current Snapshot: MvccSnapshot[applied={T|T < 7332698031958470656}]
I20260924 00:24:40.464557 17358 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:24:40.464640 17358 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:24:40.465509 17358 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad118d6f7a72487dab786802f65846b7: Sending DeleteTablet for 3 replicas of tablet 94873eb662834e098d787c1eb4015959
I20260924 00:24:40.465837 17617 tablet_service.cc:1514] Processing DeleteTablet for tablet 94873eb662834e098d787c1eb4015959 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:24:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44248
I20260924 00:24:40.465932 17483 tablet_service.cc:1514] Processing DeleteTablet for tablet 94873eb662834e098d787c1eb4015959 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:24:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56800
I20260924 00:24:40.466060 17751 tablet_service.cc:1514] Processing DeleteTablet for tablet 94873eb662834e098d787c1eb4015959 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:24:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51968
I20260924 00:24:40.466238 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 17420
I20260924 00:24:40.466262 17857 tablet_replica.cc:330] T 94873eb662834e098d787c1eb4015959 P dd7fac8544d64a7dacb21fc9ee57dc56: stopping tablet replica
I20260924 00:24:40.466348 17857 raft_consensus.cc:2229] T 94873eb662834e098d787c1eb4015959 P dd7fac8544d64a7dacb21fc9ee57dc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:24:40.466451 17857 raft_consensus.cc:2258] T 94873eb662834e098d787c1eb4015959 P dd7fac8544d64a7dacb21fc9ee57dc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:24:40.469161 17859 tablet_replica.cc:330] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627: stopping tablet replica
I20260924 00:24:40.469261 17859 raft_consensus.cc:2229] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:24:40.469400 17859 raft_consensus.cc:2258] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:24:40.469810 17859 ts_tablet_manager.cc:1905] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:24:40.469834 17857 ts_tablet_manager.cc:1905] T 94873eb662834e098d787c1eb4015959 P dd7fac8544d64a7dacb21fc9ee57dc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:24:40.471669 17857 ts_tablet_manager.cc:1918] T 94873eb662834e098d787c1eb4015959 P dd7fac8544d64a7dacb21fc9ee57dc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:24:40.471753 17857 log.cc:1196] T 94873eb662834e098d787c1eb4015959 P dd7fac8544d64a7dacb21fc9ee57dc56: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-1/wal/wals/94873eb662834e098d787c1eb4015959
I20260924 00:24:40.471972 17859 ts_tablet_manager.cc:1918] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:24:40.472039 17857 ts_tablet_manager.cc:1939] T 94873eb662834e098d787c1eb4015959 P dd7fac8544d64a7dacb21fc9ee57dc56: Deleting consensus metadata
I20260924 00:24:40.472141 17859 log.cc:1196] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/94873eb662834e098d787c1eb4015959
I20260924 00:24:40.472515 17859 ts_tablet_manager.cc:1939] T 94873eb662834e098d787c1eb4015959 P 687cca53581f4b639681c9b19ef59627: Deleting consensus metadata
I20260924 00:24:40.472548 17342 catalog_manager.cc:4841] TS dd7fac8544d64a7dacb21fc9ee57dc56 (127.0.98.2:39223): tablet 94873eb662834e098d787c1eb4015959 (table test-table [id=3cb61fed8180469ebb2f5b0fd39c6c0a]) successfully deleted
W20260924 00:24:40.472865 17345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35319 (error 108)
I20260924 00:24:40.473134 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 17554
I20260924 00:24:40.474668 17830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:40.474542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35319 (local address 127.0.0.1:56806)
0924 00:24:40.474599 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:40.474638 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 00:24:40.474890 17345 catalog_manager.cc:4584] TS a2fb45e108364c848f970a814430a48b (127.0.98.1:35319): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 94873eb662834e098d787c1eb4015959: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35319: connect: Connection refused (error 111)
I20260924 00:24:40.474946 17344 catalog_manager.cc:4841] TS 687cca53581f4b639681c9b19ef59627 (127.0.98.3:39627): tablet 94873eb662834e098d787c1eb4015959 (table test-table [id=3cb61fed8180469ebb2f5b0fd39c6c0a]) successfully deleted
I20260924 00:24:40.478366 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 17688
I20260924 00:24:40.483228 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 17326
2026-09-24T00:24:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2140 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T00:24:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:24:40Z Disabled control of system clock
I20260924 00:24:40.511485 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42351
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 00:24:40.597417 17864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:40.597646 17864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:40.597668 17864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:40.598852 17864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:24:40.598897 17864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:40.598909 17864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:40.598922 17864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:24:40.598934 17864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:24:40.600378 17864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42351
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:40.600641 17864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:40.600946 17864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:40.605914 17869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:40.606083 17872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:40.606083 17870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:40.606240 17864 server_base.cc:1034] running on GCE node
I20260924 00:24:40.606506 17864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:40.606818 17864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:40.607986 17864 hybrid_clock.cc:648] HybridClock initialized: now 1790209480607976 us; error 45 us; skew 500 ppm
I20260924 00:24:40.615492 17864 webserver.cc:460] Webserver started at http://127.0.98.62:35125/ using document root <none> and password file <none>
I20260924 00:24:40.615839 17864 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:40.615916 17864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:40.616031 17864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:40.617259 17864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "413f26ceb87148e4aa5753a299839a0e"
format_stamp: "Formatted at 2026-09-24 00:24:40 on dist-test-slave-777x"
server_key: "974b2656b894555dee5af196d796db02"
server_key_iv: "c7b40301b8ccf290f6f3acca87afe2db"
server_key_version: "encryptionkey@0"
I20260924 00:24:40.617579 17864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "413f26ceb87148e4aa5753a299839a0e"
format_stamp: "Formatted at 2026-09-24 00:24:40 on dist-test-slave-777x"
server_key: "974b2656b894555dee5af196d796db02"
server_key_iv: "c7b40301b8ccf290f6f3acca87afe2db"
server_key_version: "encryptionkey@0"
I20260924 00:24:40.618877 17864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 00:24:40.619578 17878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:40.619771 17864 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:24:40.619859 17864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "413f26ceb87148e4aa5753a299839a0e"
format_stamp: "Formatted at 2026-09-24 00:24:40 on dist-test-slave-777x"
server_key: "974b2656b894555dee5af196d796db02"
server_key_iv: "c7b40301b8ccf290f6f3acca87afe2db"
server_key_version: "encryptionkey@0"
I20260924 00:24:40.619920 17864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:40.630455 17864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:40.630709 17864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:40.630808 17864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:40.634599 17864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42351
I20260924 00:24:40.634648 17930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42351 every 8 connection(s)
I20260924 00:24:40.634963 17864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:24:40.635927 17931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:40.636337 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 17864
I20260924 00:24:40.636454 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:24:40.636653 392 external_mini_cluster.cc:1428] Setting key bd610c7c92be7f77c470dbbcfdbcf128
I20260924 00:24:40.639200 17931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e: Bootstrap starting.
I20260924 00:24:40.639874 17931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:40.640204 17931 log.cc:824] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:40.640969 17931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e: No bootstrap required, opened a new log
I20260924 00:24:40.643097 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:40.637180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54286 (local address 127.0.98.62:42351)
0924 00:24:40.637386 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:40.637393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:40.637579 (+ 186us) server_negotiation.cc:407] Connection header received
0924 00:24:40.638151 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:40.638160 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:40.638242 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:40.638420 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:40.638594 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.639924 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.640780 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.641369 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.641502 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:40.642126 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:40.642131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:40.642131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:40.642350 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:40.642394 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:40.642409 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:40.642481 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:40.642541 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 00:24:40.642621 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260924 00:24:40.642995 17931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413f26ceb87148e4aa5753a299839a0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42351 } }
I20260924 00:24:40.643162 17931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:40.643198 17931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 413f26ceb87148e4aa5753a299839a0e, State: Initialized, Role: FOLLOWER
I20260924 00:24:40.643345 17931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413f26ceb87148e4aa5753a299839a0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42351 } }
I20260924 00:24:40.643453 17931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:24:40.643486 17931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:24:40.643525 17931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:40.644160 17931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413f26ceb87148e4aa5753a299839a0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42351 } }
I20260924 00:24:40.644336 17931 leader_election.cc:302] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 413f26ceb87148e4aa5753a299839a0e; no voters:
I20260924 00:24:40.644498 17931 leader_election.cc:288] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:24:40.644546 17936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:40.644712 17936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e [term 1 LEADER]: Becoming Leader. State: Replica: 413f26ceb87148e4aa5753a299839a0e, State: Running, Role: LEADER
I20260924 00:24:40.644776 17931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:24:40.644800 17936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413f26ceb87148e4aa5753a299839a0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42351 } }
I20260924 00:24:40.645190 17938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 413f26ceb87148e4aa5753a299839a0e. Latest consensus state: current_term: 1 leader_uuid: "413f26ceb87148e4aa5753a299839a0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413f26ceb87148e4aa5753a299839a0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42351 } } }
I20260924 00:24:40.645467 17938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:40.645388 17937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "413f26ceb87148e4aa5753a299839a0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "413f26ceb87148e4aa5753a299839a0e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42351 } } }
I20260924 00:24:40.645641 17937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:40.645768 17945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:24:40.646247 17945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:24:40.647677 17945 catalog_manager.cc:1347] Generated new cluster ID: ca8a1e5d47504dc2b16cce34427ccc58
I20260924 00:24:40.647730 17945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:24:40.664430 17945 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:24:40.665171 17945 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:24:40.676661 17945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e: Generated new TSK 0
I20260924 00:24:40.676909 17945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:24:40.684894 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42351
--builtin_ntp_servers=127.0.98.20:40507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 00:24:40.768247 17955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:40.768507 17955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:40.768545 17955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:40.769732 17955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:40.769804 17955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:40.769845 17955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:24:40.771250 17955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:40.771548 17955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:40.771874 17955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:40.778030 17961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:40.778247 17960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:40.778139 17963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:40.778823 17955 server_base.cc:1034] running on GCE node
I20260924 00:24:40.779098 17955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:40.779398 17955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:40.780613 17955 hybrid_clock.cc:648] HybridClock initialized: now 1790209480780584 us; error 36 us; skew 500 ppm
I20260924 00:24:40.782191 17955 webserver.cc:460] Webserver started at http://127.0.98.1:38613/ using document root <none> and password file <none>
I20260924 00:24:40.782485 17955 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:40.782565 17955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:40.782734 17955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:40.784559 17955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "58c687ee49124ecc959f1b6d644d7799"
format_stamp: "Formatted at 2026-09-24 00:24:40 on dist-test-slave-777x"
server_key: "004e908caae9ea5a71a9ac4949b76bf7"
server_key_iv: "2cd725fa19087a1131982d7b0df14876"
server_key_version: "encryptionkey@0"
I20260924 00:24:40.784964 17955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "58c687ee49124ecc959f1b6d644d7799"
format_stamp: "Formatted at 2026-09-24 00:24:40 on dist-test-slave-777x"
server_key: "004e908caae9ea5a71a9ac4949b76bf7"
server_key_iv: "2cd725fa19087a1131982d7b0df14876"
server_key_version: "encryptionkey@0"
I20260924 00:24:40.786461 17955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:24:40.787397 17969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:40.787629 17955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:40.787750 17955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "58c687ee49124ecc959f1b6d644d7799"
format_stamp: "Formatted at 2026-09-24 00:24:40 on dist-test-slave-777x"
server_key: "004e908caae9ea5a71a9ac4949b76bf7"
server_key_iv: "2cd725fa19087a1131982d7b0df14876"
server_key_version: "encryptionkey@0"
I20260924 00:24:40.787864 17955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:40.828658 17955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:40.829041 17955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:40.829228 17955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:40.829500 17955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:24:40.829874 17955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:24:40.829933 17955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:40.829982 17955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:24:40.830010 17955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:40.836162 17955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40375
I20260924 00:24:40.836287 18082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40375 every 8 connection(s)
I20260924 00:24:40.836655 17955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:24:40.840641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 17955
I20260924 00:24:40.840787 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:24:40.840976 392 external_mini_cluster.cc:1428] Setting key 2a64baa680c3c0705b838663639d41dd
I20260924 00:24:40.842048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42351
--builtin_ntp_servers=127.0.98.20:40507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 00:24:40.842268 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:40.837537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45979 (local address 127.0.98.62:42351)
0924 00:24:40.837710 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:40.837712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:40.837775 (+ 63us) server_negotiation.cc:407] Connection header received
0924 00:24:40.838278 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:40.838279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:40.838294 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:40.838366 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:40.839069 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.839616 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.840717 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.841037 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.841117 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:40.842010 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:40.842011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:40.842012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:40.842016 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:40.842029 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:40.842033 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:40.842076 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:40.842154 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 00:24:40.842188 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260924 00:24:40.842489 18085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:40.837455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42351 (local address 127.0.98.1:45979)
0924 00:24:40.837656 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:40.837699 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:40.838171 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:40.838417 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:40.838420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:40.838428 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:24:40.838955 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:40.838960 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.839778 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.839779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:40.840623 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:40.840626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.841127 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.841128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:40.841296 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:40.841867 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:24:40.841867 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:24:40.841896 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:24:40.841919 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:24:40.842087 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:24:40.842092 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:24:40.842095 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:24:40.842159 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 00:24:40.842256 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260924 00:24:40.843004 18083 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42351
I20260924 00:24:40.843113 18083 heartbeater.cc:464] Registering TS with master...
I20260924 00:24:40.843333 18083 heartbeater.cc:511] Master 127.0.98.62:42351 requested a full tablet report, sending...
I20260924 00:24:40.843916 17895 ts_manager.cc:194] Registered new tserver with Master: 58c687ee49124ecc959f1b6d644d7799 (127.0.98.1:40375)
I20260924 00:24:40.844697 17895 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:45979
W20260924 00:24:40.924130 18086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:40.924404 18086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:40.924438 18086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:40.925614 18086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:40.925688 18086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:40.925731 18086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:24:40.927121 18086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:40.927413 18086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:40.927722 18086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:40.933300 18092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:40.933470 18086 server_base.cc:1034] running on GCE node
W20260924 00:24:40.933307 18091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:40.933300 18094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:40.933831 18086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:40.934136 18086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:40.935336 18086 hybrid_clock.cc:648] HybridClock initialized: now 1790209480935331 us; error 45 us; skew 500 ppm
I20260924 00:24:40.936695 18086 webserver.cc:460] Webserver started at http://127.0.98.2:36307/ using document root <none> and password file <none>
I20260924 00:24:40.936982 18086 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:40.937057 18086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:40.937227 18086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:40.938308 18086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "835ffec83fc141118372827b53705ddd"
format_stamp: "Formatted at 2026-09-24 00:24:40 on dist-test-slave-777x"
server_key: "929ee679a584e4ec60caac84c07e595e"
server_key_iv: "66f3727819909258b965c6981a95c58f"
server_key_version: "encryptionkey@0"
I20260924 00:24:40.938663 18086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "835ffec83fc141118372827b53705ddd"
format_stamp: "Formatted at 2026-09-24 00:24:40 on dist-test-slave-777x"
server_key: "929ee679a584e4ec60caac84c07e595e"
server_key_iv: "66f3727819909258b965c6981a95c58f"
server_key_version: "encryptionkey@0"
I20260924 00:24:40.939949 18086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 00:24:40.940711 18100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:40.940886 18086 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:40.940981 18086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "835ffec83fc141118372827b53705ddd"
format_stamp: "Formatted at 2026-09-24 00:24:40 on dist-test-slave-777x"
server_key: "929ee679a584e4ec60caac84c07e595e"
server_key_iv: "66f3727819909258b965c6981a95c58f"
server_key_version: "encryptionkey@0"
I20260924 00:24:40.941062 18086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:40.952751 18086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:40.953114 18086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:40.953302 18086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:40.953558 18086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:24:40.953914 18086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:24:40.953974 18086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:40.954012 18086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:24:40.954069 18086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:40.960891 18086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44653
I20260924 00:24:40.960927 18213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44653 every 8 connection(s)
I20260924 00:24:40.961339 18086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:24:40.966931 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:40.962182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56379 (local address 127.0.98.62:42351)
0924 00:24:40.962227 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:40.962229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:40.962423 (+ 194us) server_negotiation.cc:407] Connection header received
0924 00:24:40.962956 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:40.962958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:40.962969 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:40.963036 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:40.963808 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.964427 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.965474 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.965747 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.965824 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:40.966658 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:40.966660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:40.966660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:40.966664 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:40.966678 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:40.966682 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:40.966744 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:40.966829 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 00:24:40.966862 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:24:40.967105 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:40.962105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42351 (local address 127.0.98.2:56379)
0924 00:24:40.962296 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:40.962347 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:40.962835 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:40.963114 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:40.963118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:40.963126 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:24:40.963676 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:40.963680 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.964512 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.964513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:40.965377 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:40.965379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:40.965839 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:40.965839 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:40.965949 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:40.966511 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:24:40.966512 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:24:40.966543 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:24:40.966566 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:24:40.966753 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:24:40.966757 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:24:40.966759 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:24:40.966814 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 00:24:40.966899 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
I20260924 00:24:40.967458 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 18086
I20260924 00:24:40.967581 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:24:40.967707 18214 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42351
I20260924 00:24:40.967811 18214 heartbeater.cc:464] Registering TS with master...
I20260924 00:24:40.967835 392 external_mini_cluster.cc:1428] Setting key b8b4cc538faecec64ae086aeea547374
I20260924 00:24:40.968024 18214 heartbeater.cc:511] Master 127.0.98.62:42351 requested a full tablet report, sending...
I20260924 00:24:40.968497 17894 ts_manager.cc:194] Registered new tserver with Master: 835ffec83fc141118372827b53705ddd (127.0.98.2:44653)
I20260924 00:24:40.968914 17894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:56379
I20260924 00:24:40.968946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42351
--builtin_ntp_servers=127.0.98.20:40507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 00:24:41.051581 18217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:41.051820 18217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:41.051846 18217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:41.053058 18217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:41.053108 18217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:41.053136 18217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:24:41.054597 18217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:41.054821 18217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:41.055102 18217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:41.060351 18225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:41.060351 18223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:41.060380 18222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:41.060384 18217 server_base.cc:1034] running on GCE node
I20260924 00:24:41.060770 18217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:41.060992 18217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:41.062147 18217 hybrid_clock.cc:648] HybridClock initialized: now 1790209481062134 us; error 44 us; skew 500 ppm
I20260924 00:24:41.063601 18217 webserver.cc:460] Webserver started at http://127.0.98.3:39393/ using document root <none> and password file <none>
I20260924 00:24:41.063879 18217 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:41.063933 18217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:41.064108 18217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:41.065249 18217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "8491716d962b4e01bcf832e8090d8bc2"
format_stamp: "Formatted at 2026-09-24 00:24:41 on dist-test-slave-777x"
server_key: "0c5201ab11624da8bf21af7f0268fd67"
server_key_iv: "6855a5ab9529d53a89f31f7abd155272"
server_key_version: "encryptionkey@0"
I20260924 00:24:41.065614 18217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8491716d962b4e01bcf832e8090d8bc2"
format_stamp: "Formatted at 2026-09-24 00:24:41 on dist-test-slave-777x"
server_key: "0c5201ab11624da8bf21af7f0268fd67"
server_key_iv: "6855a5ab9529d53a89f31f7abd155272"
server_key_version: "encryptionkey@0"
I20260924 00:24:41.067009 18217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:24:41.067871 18231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:41.068050 18217 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:41.068150 18217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "8491716d962b4e01bcf832e8090d8bc2"
format_stamp: "Formatted at 2026-09-24 00:24:41 on dist-test-slave-777x"
server_key: "0c5201ab11624da8bf21af7f0268fd67"
server_key_iv: "6855a5ab9529d53a89f31f7abd155272"
server_key_version: "encryptionkey@0"
I20260924 00:24:41.068321 18217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:41.084635 18217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:41.084975 18217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:41.085131 18217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:41.085379 18217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:24:41.085762 18217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:24:41.085819 18217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:41.085872 18217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:24:41.085897 18217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:41.091960 18217 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40581
I20260924 00:24:41.092001 18344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40581 every 8 connection(s)
I20260924 00:24:41.092453 18217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:24:41.094064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 18217
I20260924 00:24:41.094166 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:24:41.094403 392 external_mini_cluster.cc:1428] Setting key 26782b813b486782950b85552842d74d
I20260924 00:24:41.099090 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:41.093319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53841 (local address 127.0.98.62:42351)
0924 00:24:41.093396 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:41.093399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:41.093867 (+ 468us) server_negotiation.cc:407] Connection header received
0924 00:24:41.094579 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:41.094581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:41.094595 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:41.094653 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:41.095641 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.096222 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.097368 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.097591 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.097691 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:41.098760 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:41.098762 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:41.098762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:41.098769 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:41.098786 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:41.098791 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:41.098847 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:41.098966 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 00:24:41.098998 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 00:24:41.099365 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:41.093322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42351 (local address 127.0.98.3:53841)
0924 00:24:41.093636 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:41.093689 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:41.094423 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:41.094743 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:41.094747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:41.094764 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:24:41.095529 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:41.095536 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.096347 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.096348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:41.097255 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:41.097258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.097703 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.097705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:41.097869 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:41.098583 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:24:41.098583 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:24:41.098617 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:24:41.098644 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:24:41.098858 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:24:41.098864 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:24:41.098868 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:24:41.098977 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 00:24:41.099103 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":93,"threads_started":1}
I20260924 00:24:41.099920 18345 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42351
I20260924 00:24:41.100037 18345 heartbeater.cc:464] Registering TS with master...
I20260924 00:24:41.100284 18345 heartbeater.cc:511] Master 127.0.98.62:42351 requested a full tablet report, sending...
I20260924 00:24:41.100699 17894 ts_manager.cc:194] Registered new tserver with Master: 8491716d962b4e01bcf832e8090d8bc2 (127.0.98.3:40581)
I20260924 00:24:41.101081 17894 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:53841
I20260924 00:24:41.106532 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:24:41.110899 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:41.107914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54292 (local address 127.0.98.62:42351)
0924 00:24:41.108006 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:41.108008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:41.108037 (+ 29us) server_negotiation.cc:407] Connection header received
0924 00:24:41.108094 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:41.108095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:41.108104 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:41.108153 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:41.108386 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.108893 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.109593 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.109769 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.109845 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:41.110556 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:41.110557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:41.110557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:41.110562 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:41.110575 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:41.110578 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:41.110659 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:41.110735 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 00:24:41.110769 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 00:24:41.112748 17894 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:24:41.114336 17894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:24:41.121018 18353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:41.116695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40375 (local address 127.0.0.1:44418)
0924 00:24:41.117040 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:41.117049 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:41.117088 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:41.117412 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:41.117415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:41.117417 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:41.117493 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:41.117496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.118880 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.118880 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:41.119917 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:41.119920 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.120686 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.120687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:41.120763 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:41.120776 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 00:24:41.120863 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 00:24:41.120919 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":37,"thread_start_us":118,"threads_started":1}
I20260924 00:24:41.121148 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:41.117131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46430 (local address 127.0.98.3:40581)
0924 00:24:41.117293 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:41.117297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:41.117305 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:24:41.117337 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:41.117338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:41.117350 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:41.117648 (+ 298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:41.117703 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.118821 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.120062 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.120811 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.120887 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:41.120972 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:41.120999 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 00:24:41.121046 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260924 00:24:41.121152 18357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:41.116925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40581 (local address 127.0.0.1:46430)
0924 00:24:41.117110 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:41.117118 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:41.117151 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:41.117536 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:41.117537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:41.117539 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:41.117574 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:41.117575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.118898 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.118899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:41.119976 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:41.119978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.120887 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.120887 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:41.120935 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:41.120937 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 00:24:41.120979 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 00:24:41.121008 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":65,"threads_started":1}
I20260924 00:24:41.121263 18354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:41.116695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44653 (local address 127.0.0.1:36662)
0924 00:24:41.116898 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:41.116938 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:41.117002 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:41.117678 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:41.117680 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:41.117682 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:41.117730 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:41.117732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.118621 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.118622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:41.119952 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:41.119954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.121048 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.121048 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:41.121100 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:41.121104 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:24:41.121153 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 00:24:41.121191 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":61,"thread_start_us":86,"threads_started":1}
I20260924 00:24:41.121779 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:41.116894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44418 (local address 127.0.98.1:40375)
0924 00:24:41.117191 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:41.117198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:41.117209 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:24:41.117254 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:41.117256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:41.117272 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:41.117386 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:41.117592 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.118768 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.120005 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.120586 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.120683 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:41.120786 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:41.121626 (+ 840us) server_negotiation.cc:299] Negotiation successful
0924 00:24:41.121670 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":74,"threads_started":1}
I20260924 00:24:41.122069 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:41.116796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36662 (local address 127.0.98.2:44653)
0924 00:24:41.117508 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:41.117512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:41.117523 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:24:41.117561 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:41.117563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:41.117579 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:41.117807 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:41.117869 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.118531 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.120060 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.120811 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.120887 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:41.120973 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:41.121944 (+ 971us) server_negotiation.cc:299] Negotiation successful
0924 00:24:41.121995 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":139,"threads_started":1}
I20260924 00:24:41.121963 18279 tablet_service.cc:1467] Processing CreateTablet for tablet fea856b809d048fab9c300cf1883394e (DEFAULT_TABLE table=test-table [id=2460188336b14d86852136645681c8b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:24:41.122370 18279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fea856b809d048fab9c300cf1883394e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:41.122470 18017 tablet_service.cc:1467] Processing CreateTablet for tablet fea856b809d048fab9c300cf1883394e (DEFAULT_TABLE table=test-table [id=2460188336b14d86852136645681c8b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:24:41.122825 18017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fea856b809d048fab9c300cf1883394e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:41.124835 18148 tablet_service.cc:1467] Processing CreateTablet for tablet fea856b809d048fab9c300cf1883394e (DEFAULT_TABLE table=test-table [id=2460188336b14d86852136645681c8b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:24:41.125155 18148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fea856b809d048fab9c300cf1883394e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:41.125551 18359 tablet_bootstrap.cc:492] T fea856b809d048fab9c300cf1883394e P 8491716d962b4e01bcf832e8090d8bc2: Bootstrap starting.
I20260924 00:24:41.126235 18359 tablet_bootstrap.cc:654] T fea856b809d048fab9c300cf1883394e P 8491716d962b4e01bcf832e8090d8bc2: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:41.126523 18359 log.cc:824] T fea856b809d048fab9c300cf1883394e P 8491716d962b4e01bcf832e8090d8bc2: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:41.126885 18360 tablet_bootstrap.cc:492] T fea856b809d048fab9c300cf1883394e P 58c687ee49124ecc959f1b6d644d7799: Bootstrap starting.
I20260924 00:24:41.127291 18359 tablet_bootstrap.cc:492] T fea856b809d048fab9c300cf1883394e P 8491716d962b4e01bcf832e8090d8bc2: No bootstrap required, opened a new log
I20260924 00:24:41.127362 18359 ts_tablet_manager.cc:1397] T fea856b809d048fab9c300cf1883394e P 8491716d962b4e01bcf832e8090d8bc2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:24:41.127550 18360 tablet_bootstrap.cc:654] T fea856b809d048fab9c300cf1883394e P 58c687ee49124ecc959f1b6d644d7799: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:41.127848 18360 log.cc:824] T fea856b809d048fab9c300cf1883394e P 58c687ee49124ecc959f1b6d644d7799: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:41.128046 18362 tablet_bootstrap.cc:492] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd: Bootstrap starting.
I20260924 00:24:41.128801 18362 tablet_bootstrap.cc:654] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:41.129004 18360 tablet_bootstrap.cc:492] T fea856b809d048fab9c300cf1883394e P 58c687ee49124ecc959f1b6d644d7799: No bootstrap required, opened a new log
I20260924 00:24:41.129079 18360 ts_tablet_manager.cc:1397] T fea856b809d048fab9c300cf1883394e P 58c687ee49124ecc959f1b6d644d7799: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:24:41.129271 18362 log.cc:824] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:41.129894 18359 raft_consensus.cc:348] T fea856b809d048fab9c300cf1883394e P 8491716d962b4e01bcf832e8090d8bc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c687ee49124ecc959f1b6d644d7799" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40375 } } peers { permanent_uuid: "835ffec83fc141118372827b53705ddd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "8491716d962b4e01bcf832e8090d8bc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40581 } }
I20260924 00:24:41.130025 18359 raft_consensus.cc:374] T fea856b809d048fab9c300cf1883394e P 8491716d962b4e01bcf832e8090d8bc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:41.130046 18359 raft_consensus.cc:729] T fea856b809d048fab9c300cf1883394e P 8491716d962b4e01bcf832e8090d8bc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8491716d962b4e01bcf832e8090d8bc2, State: Initialized, Role: FOLLOWER
I20260924 00:24:41.130129 18359 consensus_queue.cc:260] T fea856b809d048fab9c300cf1883394e P 8491716d962b4e01bcf832e8090d8bc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c687ee49124ecc959f1b6d644d7799" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40375 } } peers { permanent_uuid: "835ffec83fc141118372827b53705ddd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "8491716d962b4e01bcf832e8090d8bc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40581 } }
I20260924 00:24:41.130367 18359 ts_tablet_manager.cc:1428] T fea856b809d048fab9c300cf1883394e P 8491716d962b4e01bcf832e8090d8bc2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 00:24:41.130406 18345 heartbeater.cc:503] Master 127.0.98.62:42351 was elected leader, sending a full tablet report...
I20260924 00:24:41.130465 18362 tablet_bootstrap.cc:492] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd: No bootstrap required, opened a new log
I20260924 00:24:41.130546 18362 ts_tablet_manager.cc:1397] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:24:41.131788 18360 raft_consensus.cc:348] T fea856b809d048fab9c300cf1883394e P 58c687ee49124ecc959f1b6d644d7799 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c687ee49124ecc959f1b6d644d7799" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40375 } } peers { permanent_uuid: "835ffec83fc141118372827b53705ddd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "8491716d962b4e01bcf832e8090d8bc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40581 } }
I20260924 00:24:41.131971 18360 raft_consensus.cc:374] T fea856b809d048fab9c300cf1883394e P 58c687ee49124ecc959f1b6d644d7799 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:41.132009 18360 raft_consensus.cc:729] T fea856b809d048fab9c300cf1883394e P 58c687ee49124ecc959f1b6d644d7799 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58c687ee49124ecc959f1b6d644d7799, State: Initialized, Role: FOLLOWER
I20260924 00:24:41.132131 18360 consensus_queue.cc:260] T fea856b809d048fab9c300cf1883394e P 58c687ee49124ecc959f1b6d644d7799 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c687ee49124ecc959f1b6d644d7799" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40375 } } peers { permanent_uuid: "835ffec83fc141118372827b53705ddd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "8491716d962b4e01bcf832e8090d8bc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40581 } }
I20260924 00:24:41.132428 18360 ts_tablet_manager.cc:1428] T fea856b809d048fab9c300cf1883394e P 58c687ee49124ecc959f1b6d644d7799: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 00:24:41.132488 18083 heartbeater.cc:503] Master 127.0.98.62:42351 was elected leader, sending a full tablet report...
I20260924 00:24:41.133206 18362 raft_consensus.cc:348] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c687ee49124ecc959f1b6d644d7799" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40375 } } peers { permanent_uuid: "835ffec83fc141118372827b53705ddd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "8491716d962b4e01bcf832e8090d8bc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40581 } }
I20260924 00:24:41.133338 18362 raft_consensus.cc:374] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:41.133383 18362 raft_consensus.cc:729] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835ffec83fc141118372827b53705ddd, State: Initialized, Role: FOLLOWER
I20260924 00:24:41.133507 18362 consensus_queue.cc:260] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c687ee49124ecc959f1b6d644d7799" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40375 } } peers { permanent_uuid: "835ffec83fc141118372827b53705ddd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "8491716d962b4e01bcf832e8090d8bc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40581 } }
I20260924 00:24:41.133776 18362 ts_tablet_manager.cc:1428] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:24:41.133847 18214 heartbeater.cc:503] Master 127.0.98.62:42351 was elected leader, sending a full tablet report...
I20260924 00:24:41.167227 18367 raft_consensus.cc:482] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:24:41.167384 18367 raft_consensus.cc:504] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c687ee49124ecc959f1b6d644d7799" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40375 } } peers { permanent_uuid: "835ffec83fc141118372827b53705ddd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "8491716d962b4e01bcf832e8090d8bc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40581 } }
I20260924 00:24:41.167893 18367 leader_election.cc:288] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58c687ee49124ecc959f1b6d644d7799 (127.0.98.1:40375), 8491716d962b4e01bcf832e8090d8bc2 (127.0.98.3:40581)
I20260924 00:24:41.171008 18216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:41.168107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40581 (local address 127.0.98.2:49563)
0924 00:24:41.168157 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:41.168191 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:41.168244 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:41.168446 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:41.168448 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:41.168450 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:41.168531 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:41.168534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.169231 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.169232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:41.170071 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:41.170073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.170743 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.170744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:41.170811 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:41.170814 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:24:41.170879 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 00:24:41.170927 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 00:24:41.171008 18358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:41.168232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49563 (local address 127.0.98.3:40581)
0924 00:24:41.168279 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:41.168282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:41.168291 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:24:41.168323 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:41.168324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:41.168334 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:41.168403 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:41.168614 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.169139 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.170168 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.170617 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.170714 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:41.170726 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:41.170897 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 00:24:41.170941 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 00:24:41.171151 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:41.168102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40375 (local address 127.0.98.2:36865)
0924 00:24:41.168363 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:41.168372 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:41.168422 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:41.168596 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:41.168596 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:41.168598 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:41.168670 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:41.168671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.169416 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.169416 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:41.170192 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:41.170194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.170875 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.170876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:41.170952 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:41.170955 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:24:41.171024 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 00:24:41.171067 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":33,"thread_start_us":132,"threads_started":1}
I20260924 00:24:41.171137 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:41.168254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36865 (local address 127.0.98.1:40375)
0924 00:24:41.168309 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:41.168311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:41.168437 (+ 126us) server_negotiation.cc:407] Connection header received
0924 00:24:41.168472 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:41.168473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:41.168486 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:41.168565 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:41.168770 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.169330 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.170286 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.170770 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.170869 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:41.170881 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:41.171030 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 00:24:41.171071 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 00:24:41.171499 18299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea856b809d048fab9c300cf1883394e" candidate_uuid: "835ffec83fc141118372827b53705ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8491716d962b4e01bcf832e8090d8bc2" is_pre_election: true
I20260924 00:24:41.171512 18037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea856b809d048fab9c300cf1883394e" candidate_uuid: "835ffec83fc141118372827b53705ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58c687ee49124ecc959f1b6d644d7799" is_pre_election: true
I20260924 00:24:41.171657 18299 raft_consensus.cc:2454] T fea856b809d048fab9c300cf1883394e P 8491716d962b4e01bcf832e8090d8bc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835ffec83fc141118372827b53705ddd in term 0.
I20260924 00:24:41.171657 18037 raft_consensus.cc:2454] T fea856b809d048fab9c300cf1883394e P 58c687ee49124ecc959f1b6d644d7799 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835ffec83fc141118372827b53705ddd in term 0.
I20260924 00:24:41.171936 18101 leader_election.cc:302] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835ffec83fc141118372827b53705ddd, 8491716d962b4e01bcf832e8090d8bc2; no voters:
I20260924 00:24:41.172078 18367 raft_consensus.cc:2781] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:24:41.172137 18367 raft_consensus.cc:482] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:24:41.172225 18367 raft_consensus.cc:3037] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:41.172945 18367 raft_consensus.cc:504] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c687ee49124ecc959f1b6d644d7799" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40375 } } peers { permanent_uuid: "835ffec83fc141118372827b53705ddd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "8491716d962b4e01bcf832e8090d8bc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40581 } }
I20260924 00:24:41.173095 18367 leader_election.cc:288] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd [CANDIDATE]: Term 1 election: Requested vote from peers 58c687ee49124ecc959f1b6d644d7799 (127.0.98.1:40375), 8491716d962b4e01bcf832e8090d8bc2 (127.0.98.3:40581)
I20260924 00:24:41.173326 18037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea856b809d048fab9c300cf1883394e" candidate_uuid: "835ffec83fc141118372827b53705ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58c687ee49124ecc959f1b6d644d7799"
I20260924 00:24:41.173298 18299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fea856b809d048fab9c300cf1883394e" candidate_uuid: "835ffec83fc141118372827b53705ddd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8491716d962b4e01bcf832e8090d8bc2"
I20260924 00:24:41.173409 18037 raft_consensus.cc:3037] T fea856b809d048fab9c300cf1883394e P 58c687ee49124ecc959f1b6d644d7799 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:41.173429 18299 raft_consensus.cc:3037] T fea856b809d048fab9c300cf1883394e P 8491716d962b4e01bcf832e8090d8bc2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:41.174222 18299 raft_consensus.cc:2454] T fea856b809d048fab9c300cf1883394e P 8491716d962b4e01bcf832e8090d8bc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835ffec83fc141118372827b53705ddd in term 1.
I20260924 00:24:41.174247 18037 raft_consensus.cc:2454] T fea856b809d048fab9c300cf1883394e P 58c687ee49124ecc959f1b6d644d7799 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835ffec83fc141118372827b53705ddd in term 1.
I20260924 00:24:41.174413 18101 leader_election.cc:302] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835ffec83fc141118372827b53705ddd, 8491716d962b4e01bcf832e8090d8bc2; no voters:
I20260924 00:24:41.174543 18367 raft_consensus.cc:2781] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:41.174741 18367 raft_consensus.cc:686] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd [term 1 LEADER]: Becoming Leader. State: Replica: 835ffec83fc141118372827b53705ddd, State: Running, Role: LEADER
I20260924 00:24:41.174849 18367 consensus_queue.cc:237] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c687ee49124ecc959f1b6d644d7799" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40375 } } peers { permanent_uuid: "835ffec83fc141118372827b53705ddd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "8491716d962b4e01bcf832e8090d8bc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40581 } }
I20260924 00:24:41.175722 17895 catalog_manager.cc:5495] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd reported cstate change: term changed from 0 to 1, leader changed from <none> to 835ffec83fc141118372827b53705ddd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "835ffec83fc141118372827b53705ddd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58c687ee49124ecc959f1b6d644d7799" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835ffec83fc141118372827b53705ddd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8491716d962b4e01bcf832e8090d8bc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40581 } health_report { overall_health: UNKNOWN } } }
I20260924 00:24:41.195760 18355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:41.193057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36672 (local address 127.0.98.2:44653)
0924 00:24:41.193105 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:41.193108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:41.193118 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:24:41.193219 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:41.193220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:41.193242 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:41.193333 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:24:41.193509 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.194032 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.194875 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:41.195096 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:41.195185 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:41.195362 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:24:41.195466 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:24:41.195592 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:41.195627 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 00:24:41.195636 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:24:41.197772 18037 raft_consensus.cc:1261] T fea856b809d048fab9c300cf1883394e P 58c687ee49124ecc959f1b6d644d7799 [term 1 FOLLOWER]: Refusing update from remote peer 835ffec83fc141118372827b53705ddd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:24:41.197772 18299 raft_consensus.cc:1261] T fea856b809d048fab9c300cf1883394e P 8491716d962b4e01bcf832e8090d8bc2 [term 1 FOLLOWER]: Refusing update from remote peer 835ffec83fc141118372827b53705ddd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:24:41.198184 18367 consensus_queue.cc:1035] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "58c687ee49124ecc959f1b6d644d7799" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:24:41.198357 18369 consensus_queue.cc:1035] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8491716d962b4e01bcf832e8090d8bc2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:24:41.200877 18372 mvcc.cc:205] Tried to move back new op lower bound from 7332698034983182336 to 7332698034892947456. Current Snapshot: MvccSnapshot[applied={T|T < 7332698034983182336}]
I20260924 00:24:41.201640 18374 mvcc.cc:205] Tried to move back new op lower bound from 7332698034983182336 to 7332698034892947456. Current Snapshot: MvccSnapshot[applied={T|T < 7332698034983182336}]
I20260924 00:24:41.203243 17894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:24:41.203349 17894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:24:41.204270 17894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 413f26ceb87148e4aa5753a299839a0e: Sending DeleteTablet for 3 replicas of tablet fea856b809d048fab9c300cf1883394e
I20260924 00:24:41.204622 18017 tablet_service.cc:1514] Processing DeleteTablet for tablet fea856b809d048fab9c300cf1883394e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:24:41 UTC) from {username='slave'} at 127.0.0.1:44418
I20260924 00:24:41.204638 18279 tablet_service.cc:1514] Processing DeleteTablet for tablet fea856b809d048fab9c300cf1883394e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:24:41 UTC) from {username='slave'} at 127.0.0.1:46430
I20260924 00:24:41.204921 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 17955
I20260924 00:24:41.204959 18148 tablet_service.cc:1514] Processing DeleteTablet for tablet fea856b809d048fab9c300cf1883394e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:24:41 UTC) from {username='slave'} at 127.0.0.1:36662
I20260924 00:24:41.208333 18384 tablet_replica.cc:330] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd: stopping tablet replica
I20260924 00:24:41.208463 18384 raft_consensus.cc:2229] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:24:41.208576 18384 raft_consensus.cc:2258] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:24:41.208758 18383 tablet_replica.cc:330] T fea856b809d048fab9c300cf1883394e P 8491716d962b4e01bcf832e8090d8bc2: stopping tablet replica
I20260924 00:24:41.208866 18383 raft_consensus.cc:2229] T fea856b809d048fab9c300cf1883394e P 8491716d962b4e01bcf832e8090d8bc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:24:41.208976 18383 raft_consensus.cc:2258] T fea856b809d048fab9c300cf1883394e P 8491716d962b4e01bcf832e8090d8bc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:24:41.209241 18384 ts_tablet_manager.cc:1905] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:24:41.209472 18383 ts_tablet_manager.cc:1905] T fea856b809d048fab9c300cf1883394e P 8491716d962b4e01bcf832e8090d8bc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:24:41.211516 18384 ts_tablet_manager.cc:1918] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:24:41.211699 18384 log.cc:1196] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-1/wal/wals/fea856b809d048fab9c300cf1883394e
I20260924 00:24:41.211690 18383 ts_tablet_manager.cc:1918] T fea856b809d048fab9c300cf1883394e P 8491716d962b4e01bcf832e8090d8bc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:24:41.211768 18383 log.cc:1196] T fea856b809d048fab9c300cf1883394e P 8491716d962b4e01bcf832e8090d8bc2: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/fea856b809d048fab9c300cf1883394e
W20260924 00:24:41.211977 17882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40375 (error 108)
I20260924 00:24:41.212055 18383 ts_tablet_manager.cc:1939] T fea856b809d048fab9c300cf1883394e P 8491716d962b4e01bcf832e8090d8bc2: Deleting consensus metadata
I20260924 00:24:41.212332 18384 ts_tablet_manager.cc:1939] T fea856b809d048fab9c300cf1883394e P 835ffec83fc141118372827b53705ddd: Deleting consensus metadata
I20260924 00:24:41.212576 17879 catalog_manager.cc:4841] TS 8491716d962b4e01bcf832e8090d8bc2 (127.0.98.3:40581): tablet fea856b809d048fab9c300cf1883394e (table test-table [id=2460188336b14d86852136645681c8b4]) successfully deleted
I20260924 00:24:41.212808 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 18086
I20260924 00:24:41.213492 18354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:41.212916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40375 (local address 127.0.0.1:44432)
0924 00:24:41.213402 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:41.213449 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462}
W20260924 00:24:41.213678 17882 catalog_manager.cc:4584] TS 58c687ee49124ecc959f1b6d644d7799 (127.0.98.1:40375): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fea856b809d048fab9c300cf1883394e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40375: connect: Connection refused (error 111)
I20260924 00:24:41.213827 17879 catalog_manager.cc:4841] TS 835ffec83fc141118372827b53705ddd (127.0.98.2:44653): tablet fea856b809d048fab9c300cf1883394e (table test-table [id=2460188336b14d86852136645681c8b4]) successfully deleted
I20260924 00:24:41.217866 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 18217
I20260924 00:24:41.222934 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 17864
2026-09-24T00:24:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (739 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:24:41 dist-test-slave-777x krb5kdc[18388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:24:41 dist-test-slave-777x krb5kdc[18388](info): set up 2 sockets
Sep 24 00:24:41 dist-test-slave-777x krb5kdc[18388](info): commencing operation
krb5kdc: starting...
W20260924 00:24:42.272061 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:24:42 dist-test-slave-777x krb5kdc[18388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:24:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:24:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:24:42.400045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43521
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:42.483789 18401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:42.484035 18401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:42.484073 18401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:42.485286 18401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:24:42.485355 18401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:42.485378 18401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:42.485399 18401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:24:42.485428 18401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:24:42.486932 18401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:42.487226 18401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:42.487453 18401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:42.492646 18409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:42.492635 18406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:42.492631 18407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:42.492832 18401 server_base.cc:1034] running on GCE node
I20260924 00:24:42.493213 18401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:42.493548 18401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:42.494724 18401 hybrid_clock.cc:648] HybridClock initialized: now 1790209482494704 us; error 33 us; skew 500 ppm
Sep 24 00:24:42 dist-test-slave-777x krb5kdc[18388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:42.497570 18401 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:24:42.498131 18401 webserver.cc:460] Webserver started at http://127.0.98.62:41159/ using document root <none> and password file <none>
I20260924 00:24:42.498374 18401 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:42.498420 18401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:42.498582 18401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:42.499758 18401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "5e4ca6d4098b41849dc803e9cd80c64d"
format_stamp: "Formatted at 2026-09-24 00:24:42 on dist-test-slave-777x"
server_key: "343b5b89184d17ea2487458588dd2b78"
server_key_iv: "7297eaca9f86afd2be34832b7d921913"
server_key_version: "encryptionkey@0"
I20260924 00:24:42.500136 18401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "5e4ca6d4098b41849dc803e9cd80c64d"
format_stamp: "Formatted at 2026-09-24 00:24:42 on dist-test-slave-777x"
server_key: "343b5b89184d17ea2487458588dd2b78"
server_key_iv: "7297eaca9f86afd2be34832b7d921913"
server_key_version: "encryptionkey@0"
I20260924 00:24:42.501576 18401 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:42.502565 18416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:42.502825 18401 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:42.502947 18401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "5e4ca6d4098b41849dc803e9cd80c64d"
format_stamp: "Formatted at 2026-09-24 00:24:42 on dist-test-slave-777x"
server_key: "343b5b89184d17ea2487458588dd2b78"
server_key_iv: "7297eaca9f86afd2be34832b7d921913"
server_key_version: "encryptionkey@0"
I20260924 00:24:42.503039 18401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:42.515725 18401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:42.518049 18401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:42.518168 18401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:42.522223 18401 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43521
I20260924 00:24:42.522282 18468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43521 every 8 connection(s)
I20260924 00:24:42.522661 18401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:24:42.523600 18469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:42.525004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 18401
I20260924 00:24:42.525164 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:24:42.525393 392 external_mini_cluster.cc:1428] Setting key 1e1171a332673dc00ead6fafa2f70152
I20260924 00:24:42.526846 18469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d: Bootstrap starting.
I20260924 00:24:42.527516 18469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:42.527817 18469 log.cc:824] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:42.528589 18469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d: No bootstrap required, opened a new log
I20260924 00:24:42.530395 18469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4ca6d4098b41849dc803e9cd80c64d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43521 } }
I20260924 00:24:42.530597 18469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:42.530638 18469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e4ca6d4098b41849dc803e9cd80c64d, State: Initialized, Role: FOLLOWER
I20260924 00:24:42.530771 18469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4ca6d4098b41849dc803e9cd80c64d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43521 } }
I20260924 00:24:42.530869 18469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:24:42.530921 18469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:24:42.530979 18469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:42.531589 18469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4ca6d4098b41849dc803e9cd80c64d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43521 } }
I20260924 00:24:42.531716 18469 leader_election.cc:302] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e4ca6d4098b41849dc803e9cd80c64d; no voters:
I20260924 00:24:42.531917 18469 leader_election.cc:288] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:24:42.531980 18474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:42.532166 18474 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d [term 1 LEADER]: Becoming Leader. State: Replica: 5e4ca6d4098b41849dc803e9cd80c64d, State: Running, Role: LEADER
I20260924 00:24:42.532253 18469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:24:42.532302 18474 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4ca6d4098b41849dc803e9cd80c64d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43521 } }
Sep 24 00:24:42 dist-test-slave-777x krb5kdc[18388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:24:42.532639 18474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e4ca6d4098b41849dc803e9cd80c64d. Latest consensus state: current_term: 1 leader_uuid: "5e4ca6d4098b41849dc803e9cd80c64d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4ca6d4098b41849dc803e9cd80c64d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43521 } } }
I20260924 00:24:42.532727 18474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:42.532649 18476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e4ca6d4098b41849dc803e9cd80c64d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4ca6d4098b41849dc803e9cd80c64d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43521 } } }
I20260924 00:24:42.532940 18480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:24:42.532984 18476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:42.533494 18480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:24:42.535590 18480 catalog_manager.cc:1347] Generated new cluster ID: fb2975cdbc964ab88f111715a67bfdf8
I20260924 00:24:42.535645 18480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:24:42.537328 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:42.525988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54508 (local address 127.0.98.62:43521)
0924 00:24:42.526232 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:42.526242 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:42.526280 (+ 38us) server_negotiation.cc:407] Connection header received
0924 00:24:42.526833 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:42.526852 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:42.526928 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:42.527080 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:42.527809 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:42.528904 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:42.529707 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:42.530125 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:42.530280 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:42.533277 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:42.533287 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:42.533288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:42.533303 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:42.535507 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:42.535977 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:42.535981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:42.535982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:42.536096 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:42.536393 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:42.536395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:42.536395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:42.536657 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:42.536844 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:42.536957 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 00:24:42.537103 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":108,"threads_started":1}
I20260924 00:24:42.557598 18480 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:24:42.558205 18480 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:24:42.570196 18480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d: Generated new TSK 0
I20260924 00:24:42.570474 18480 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 00:24:42.609302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43521
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:42.706712 18495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:42.706936 18495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:42.706964 18495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:42.708266 18495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:42.708316 18495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:42.708369 18495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:24:42.710095 18495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:42.710366 18495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:42.710616 18495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:42.715893 18501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:42.715905 18500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:42.715919 18503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:42.716151 18495 server_base.cc:1034] running on GCE node
I20260924 00:24:42.716413 18495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:42.716686 18495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:42.717860 18495 hybrid_clock.cc:648] HybridClock initialized: now 1790209482717851 us; error 41 us; skew 500 ppm
Sep 24 00:24:42 dist-test-slave-777x krb5kdc[18388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:42.720444 18495 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 00:24:42.720954 18495 webserver.cc:460] Webserver started at http://127.0.98.1:44229/ using document root <none> and password file <none>
I20260924 00:24:42.721189 18495 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:42.721253 18495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:42.721383 18495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:42.722453 18495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "75f71cbdbd9f451db188a60292d52bf8"
format_stamp: "Formatted at 2026-09-24 00:24:42 on dist-test-slave-777x"
server_key: "a0ad139481eb71f2f9a94dc60b0718d9"
server_key_iv: "db81666767994c992ddc79da86d97d8e"
server_key_version: "encryptionkey@0"
I20260924 00:24:42.722821 18495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "75f71cbdbd9f451db188a60292d52bf8"
format_stamp: "Formatted at 2026-09-24 00:24:42 on dist-test-slave-777x"
server_key: "a0ad139481eb71f2f9a94dc60b0718d9"
server_key_iv: "db81666767994c992ddc79da86d97d8e"
server_key_version: "encryptionkey@0"
I20260924 00:24:42.724309 18495 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 00:24:42.725159 18510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:42.725327 18495 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:42.725425 18495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "75f71cbdbd9f451db188a60292d52bf8"
format_stamp: "Formatted at 2026-09-24 00:24:42 on dist-test-slave-777x"
server_key: "a0ad139481eb71f2f9a94dc60b0718d9"
server_key_iv: "db81666767994c992ddc79da86d97d8e"
server_key_version: "encryptionkey@0"
I20260924 00:24:42.725504 18495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:42.742431 18495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:42.744467 18495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:42.744637 18495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:42.744882 18495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:24:42.745182 18495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:24:42.745234 18495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:42.745270 18495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:24:42.745309 18495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:42.750910 18495 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36745
I20260924 00:24:42.750959 18623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36745 every 8 connection(s)
I20260924 00:24:42.751282 18495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:24:42.755488 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 18495
I20260924 00:24:42.755646 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:24:42.755987 392 external_mini_cluster.cc:1428] Setting key 8a8739beabc15bd8d38367ec212d32f3
Sep 24 00:24:42 dist-test-slave-777x krb5kdc[18388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:24:42.762362 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:42.752009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41211 (local address 127.0.98.62:43521)
0924 00:24:42.752090 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:42.752093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:42.752351 (+ 258us) server_negotiation.cc:407] Connection header received
0924 00:24:42.753251 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:42.753252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:42.753264 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:42.753321 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:42.753951 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:42.754526 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:42.755572 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:42.755957 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:42.756049 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:42.758846 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:42.758848 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:42.758848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:42.758859 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:42.760703 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:42.761223 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:42.761229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:42.761229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:42.761289 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:42.761618 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:42.761619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:42.761619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:42.761748 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:42.761801 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:42.762207 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 00:24:42.762294 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 00:24:42.762774 18626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:42.751988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43521 (local address 127.0.98.1:41211)
0924 00:24:42.752196 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:42.752242 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:42.753150 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:42.753383 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:42.753387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:42.753647 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:24:42.753850 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:42.753854 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:42.754613 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:42.754613 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:42.755466 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:42.755468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:42.756102 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:42.756104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:42.756313 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:42.756951 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:24:42.756960 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:24:42.758737 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:24:42.760783 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:24:42.760792 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:24:42.760794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:24:42.761144 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:24:42.761355 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:24:42.761355 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:24:42.761356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:24:42.761515 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:24:42.761814 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:24:42.761822 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:24:42.762018 (+ 196us) client_negotiation.cc:769] Sending connection context
0924 00:24:42.762155 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 00:24:42.762385 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":103,"threads_started":1}
I20260924 00:24:42.763326 18624 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43521
I20260924 00:24:42.763450 18624 heartbeater.cc:464] Registering TS with master...
I20260924 00:24:42.763669 18624 heartbeater.cc:511] Master 127.0.98.62:43521 requested a full tablet report, sending...
I20260924 00:24:42.764240 18433 ts_manager.cc:194] Registered new tserver with Master: 75f71cbdbd9f451db188a60292d52bf8 (127.0.98.1:36745)
I20260924 00:24:42.764854 18433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41211
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 00:24:42.784649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43521
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:42.867455 18629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:42.867727 18629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:42.867764 18629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:42.869022 18629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:42.869091 18629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:42.869133 18629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:24:42.870633 18629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:42.870899 18629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:42.871120 18629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:24:42.876252 18629 server_base.cc:1034] running on GCE node
W20260924 00:24:42.876156 18637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:42.876191 18634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:42.876201 18635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:42.876621 18629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:42.876906 18629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:42.878072 18629 hybrid_clock.cc:648] HybridClock initialized: now 1790209482878051 us; error 36 us; skew 500 ppm
Sep 24 00:24:42 dist-test-slave-777x krb5kdc[18388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:42.880868 18629 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 00:24:42.881398 18629 webserver.cc:460] Webserver started at http://127.0.98.2:42211/ using document root <none> and password file <none>
I20260924 00:24:42.881657 18629 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:42.881708 18629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:42.881876 18629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:42.883040 18629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "e81368c22024482b9256b728cca3b62c"
format_stamp: "Formatted at 2026-09-24 00:24:42 on dist-test-slave-777x"
server_key: "7f5221d53f7ed6c4cbcb91ba788946bb"
server_key_iv: "1352bd4175e3c3001267c71697be973f"
server_key_version: "encryptionkey@0"
I20260924 00:24:42.883450 18629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e81368c22024482b9256b728cca3b62c"
format_stamp: "Formatted at 2026-09-24 00:24:42 on dist-test-slave-777x"
server_key: "7f5221d53f7ed6c4cbcb91ba788946bb"
server_key_iv: "1352bd4175e3c3001267c71697be973f"
server_key_version: "encryptionkey@0"
I20260924 00:24:42.884936 18629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:24:42.885779 18644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:42.885977 18629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:42.886065 18629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "e81368c22024482b9256b728cca3b62c"
format_stamp: "Formatted at 2026-09-24 00:24:42 on dist-test-slave-777x"
server_key: "7f5221d53f7ed6c4cbcb91ba788946bb"
server_key_iv: "1352bd4175e3c3001267c71697be973f"
server_key_version: "encryptionkey@0"
I20260924 00:24:42.886139 18629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:42.914660 18629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:42.917229 18629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:42.917404 18629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:42.917680 18629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:24:42.918013 18629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:24:42.918048 18629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:42.918071 18629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:24:42.918116 18629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:42.924160 18629 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33927
I20260924 00:24:42.924263 18757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33927 every 8 connection(s)
I20260924 00:24:42.924579 18629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:24:42.930030 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 18629
I20260924 00:24:42.930187 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:24:42.930424 392 external_mini_cluster.cc:1428] Setting key 55780bff1554fceee1e1bb9052a36c91
Sep 24 00:24:42 dist-test-slave-777x krb5kdc[18388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:24:42.936280 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:42.925355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59269 (local address 127.0.98.62:43521)
0924 00:24:42.925437 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:42.925439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:42.925623 (+ 184us) server_negotiation.cc:407] Connection header received
0924 00:24:42.926620 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:42.926621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:42.926632 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:42.926700 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:42.927377 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:42.927979 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:42.929162 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:42.929380 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:42.929521 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:42.932509 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:42.932511 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:42.932512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:42.932520 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:42.934463 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:42.935028 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:42.935029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:42.935029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:42.935085 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:42.935415 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:42.935417 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:42.935417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:42.935583 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:42.935649 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:42.936057 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 00:24:42.936135 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 00:24:42.936551 18760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:42.925328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43521 (local address 127.0.98.2:59269)
0924 00:24:42.925490 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:42.925533 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:42.926503 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:42.926793 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:42.926797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:42.927068 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:24:42.927274 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:42.927277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:42.928070 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:42.928070 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:42.929060 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:42.929063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:42.929540 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:42.929541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:42.929649 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:42.930289 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:24:42.930301 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:24:42.932410 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:24:42.934550 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:24:42.934560 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:24:42.934563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:24:42.934937 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:24:42.935154 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:24:42.935155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:24:42.935155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:24:42.935317 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:24:42.935659 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:24:42.935667 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:24:42.935873 (+ 206us) client_negotiation.cc:769] Sending connection context
0924 00:24:42.936024 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 00:24:42.936254 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260924 00:24:42.937103 18758 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43521
I20260924 00:24:42.937214 18758 heartbeater.cc:464] Registering TS with master...
I20260924 00:24:42.937434 18758 heartbeater.cc:511] Master 127.0.98.62:43521 requested a full tablet report, sending...
I20260924 00:24:42.937809 18433 ts_manager.cc:194] Registered new tserver with Master: e81368c22024482b9256b728cca3b62c (127.0.98.2:33927)
I20260924 00:24:42.938185 18433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59269
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 00:24:42.959859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43521
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:43.046150 18763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:43.046363 18763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:43.046394 18763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:43.047608 18763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:43.047657 18763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:43.047683 18763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:24:43.049227 18763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:43.049484 18763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:43.049717 18763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:43.054865 18771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:43.054904 18769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:43.054868 18768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:43.055286 18763 server_base.cc:1034] running on GCE node
I20260924 00:24:43.055540 18763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:43.055802 18763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:43.056977 18763 hybrid_clock.cc:648] HybridClock initialized: now 1790209483056958 us; error 26 us; skew 500 ppm
Sep 24 00:24:43 dist-test-slave-777x krb5kdc[18388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:43.059702 18763 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 00:24:43.060225 18763 webserver.cc:460] Webserver started at http://127.0.98.3:44271/ using document root <none> and password file <none>
I20260924 00:24:43.060501 18763 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:43.060573 18763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:43.060753 18763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:43.062029 18763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "b7aeebd0e01f424b9bdc66b39bac7602"
format_stamp: "Formatted at 2026-09-24 00:24:43 on dist-test-slave-777x"
server_key: "8e49836ed399816e8a1c5778aedf231d"
server_key_iv: "6246e726a6c2c9e5d34f765a236d7c3c"
server_key_version: "encryptionkey@0"
I20260924 00:24:43.062426 18763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b7aeebd0e01f424b9bdc66b39bac7602"
format_stamp: "Formatted at 2026-09-24 00:24:43 on dist-test-slave-777x"
server_key: "8e49836ed399816e8a1c5778aedf231d"
server_key_iv: "6246e726a6c2c9e5d34f765a236d7c3c"
server_key_version: "encryptionkey@0"
I20260924 00:24:43.063930 18763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:24:43.064957 18778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:43.065186 18763 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:24:43.065295 18763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "b7aeebd0e01f424b9bdc66b39bac7602"
format_stamp: "Formatted at 2026-09-24 00:24:43 on dist-test-slave-777x"
server_key: "8e49836ed399816e8a1c5778aedf231d"
server_key_iv: "6246e726a6c2c9e5d34f765a236d7c3c"
server_key_version: "encryptionkey@0"
I20260924 00:24:43.065371 18763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:43.083352 18763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:43.085644 18763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:43.085876 18763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:43.086153 18763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:24:43.086465 18763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:24:43.086514 18763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:43.086555 18763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:24:43.086588 18763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:43.092567 18763 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43979
I20260924 00:24:43.092617 18891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43979 every 8 connection(s)
I20260924 00:24:43.092957 18763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:24:43.095055 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 18763
I20260924 00:24:43.095162 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:24:43.095356 392 external_mini_cluster.cc:1428] Setting key a463a944f9b3ab44a0367d5284f50937
Sep 24 00:24:43 dist-test-slave-777x krb5kdc[18388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209483, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:24:43.102710 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.093663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60315 (local address 127.0.98.62:43521)
0924 00:24:43.093715 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:43.093717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:43.093989 (+ 272us) server_negotiation.cc:407] Connection header received
0924 00:24:43.094868 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:43.094869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:43.094879 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:43.094935 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:43.095733 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.096548 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.097533 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.097713 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.097777 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.099985 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:43.099987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:43.099987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:43.099993 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:43.101493 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:43.101879 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:43.101880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:43.101880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:43.101939 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:43.102145 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:43.102146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:43.102146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:43.102274 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:43.102326 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:43.102559 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 00:24:43.102630 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:24:43.102854 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.093631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43521 (local address 127.0.98.3:60315)
0924 00:24:43.093790 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:43.093832 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:43.094769 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:43.095015 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:43.095020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:43.095412 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:24:43.095665 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.095670 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.096624 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.096625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.097451 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.097454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.097795 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.097796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.097922 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.098452 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:24:43.098461 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:24:43.099889 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:24:43.101575 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:24:43.101581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:24:43.101583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:24:43.101813 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:24:43.101997 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:24:43.101998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:24:43.101998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:24:43.102094 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:24:43.102325 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:24:43.102329 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:24:43.102454 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 00:24:43.102537 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 00:24:43.102654 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260924 00:24:43.103312 18892 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43521
I20260924 00:24:43.103406 18892 heartbeater.cc:464] Registering TS with master...
I20260924 00:24:43.103569 18892 heartbeater.cc:511] Master 127.0.98.62:43521 requested a full tablet report, sending...
I20260924 00:24:43.103957 18433 ts_manager.cc:194] Registered new tserver with Master: b7aeebd0e01f424b9bdc66b39bac7602 (127.0.98.3:43979)
I20260924 00:24:43.104355 18433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60315
I20260924 00:24:43.107223 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:24:43.114377 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.108550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54510 (local address 127.0.98.62:43521)
0924 00:24:43.108593 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:43.108595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:43.108710 (+ 115us) server_negotiation.cc:407] Connection header received
0924 00:24:43.108801 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:43.108802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:43.108811 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:43.108858 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:43.109396 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.109885 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.110555 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.110730 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.110811 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.111972 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:43.111974 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:43.111974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:43.111981 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:43.113222 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:43.113577 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:43.113578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:43.113579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:43.113632 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:43.113824 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:43.113825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:43.113825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:43.113983 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:43.114096 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:43.114200 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 00:24:43.114275 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:24:43.115845 18433 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:24:43.117496 18433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:24:43.124828 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.120425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49118 (local address 127.0.98.3:43979)
0924 00:24:43.120607 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:43.120612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:43.121279 (+ 667us) server_negotiation.cc:407] Connection header received
0924 00:24:43.121331 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:43.121333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:43.121349 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:43.121425 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:43.121541 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.122552 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.123619 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.124357 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.124436 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.124499 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:43.124692 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 00:24:43.124743 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":86,"threads_started":1}
I20260924 00:24:43.124812 18905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.120136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43979 (local address 127.0.0.1:49118)
0924 00:24:43.121153 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:43.121164 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:43.121215 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:43.121433 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:43.121434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:43.121436 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:43.121476 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.121478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.122648 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.122649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.123538 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.123541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.124469 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.124470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.124563 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.124578 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 00:24:43.124655 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 00:24:43.124693 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"mutex_wait_us":4,"thread_start_us":158,"threads_started":1}
I20260924 00:24:43.124812 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.120134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36745 (local address 127.0.0.1:36512)
0924 00:24:43.120300 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:43.120341 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:43.120406 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:43.121051 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:43.121053 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:43.121055 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:43.121100 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.121102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.122550 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.122551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.123706 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.123708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.124471 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.124472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.124563 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.124578 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 00:24:43.124632 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 00:24:43.124696 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":5,"thread_start_us":61,"threads_started":1}
I20260924 00:24:43.124907 18907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.120408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33898 (local address 127.0.98.2:33927)
0924 00:24:43.120587 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:43.120597 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:43.120606 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:24:43.120637 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:43.120639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:43.120651 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:43.120733 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:43.120913 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.122132 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.123765 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.124474 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.124540 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.124575 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:43.124774 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 00:24:43.124814 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260924 00:24:43.125133 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.120136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33927 (local address 127.0.0.1:33898)
0924 00:24:43.120441 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:43.120450 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:43.120491 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:43.120743 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:43.120747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:43.120749 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:43.120826 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.120829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.122278 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.122279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.123665 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.123668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.124603 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.124604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.124681 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.124686 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:24:43.124978 (+ 292us) client_negotiation.cc:240] Negotiation successful
0924 00:24:43.125039 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":105,"threads_started":1}
I20260924 00:24:43.125514 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.120257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36512 (local address 127.0.98.1:36745)
0924 00:24:43.120892 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:43.120896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:43.120907 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:24:43.120943 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:43.120945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:43.120960 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:43.121040 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:43.121181 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.122441 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.123805 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.124363 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.124851 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.125111 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:43.125140 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 00:24:43.125415 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":84,"threads_started":1}
I20260924 00:24:43.125907 18826 tablet_service.cc:1467] Processing CreateTablet for tablet 530be28e16ee4f899302b84dda4436aa (DEFAULT_TABLE table=test-table [id=98327fe20d7b4fd0ad53a27ffe86c3a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:24:43.126113 18558 tablet_service.cc:1467] Processing CreateTablet for tablet 530be28e16ee4f899302b84dda4436aa (DEFAULT_TABLE table=test-table [id=98327fe20d7b4fd0ad53a27ffe86c3a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:24:43.126114 18692 tablet_service.cc:1467] Processing CreateTablet for tablet 530be28e16ee4f899302b84dda4436aa (DEFAULT_TABLE table=test-table [id=98327fe20d7b4fd0ad53a27ffe86c3a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:24:43.126260 18826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 530be28e16ee4f899302b84dda4436aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:43.126454 18692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 530be28e16ee4f899302b84dda4436aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:43.126454 18558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 530be28e16ee4f899302b84dda4436aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:43.129237 18909 tablet_bootstrap.cc:492] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602: Bootstrap starting.
I20260924 00:24:43.129850 18909 tablet_bootstrap.cc:654] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:43.130002 18910 tablet_bootstrap.cc:492] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c: Bootstrap starting.
I20260924 00:24:43.130175 18911 tablet_bootstrap.cc:492] T 530be28e16ee4f899302b84dda4436aa P 75f71cbdbd9f451db188a60292d52bf8: Bootstrap starting.
I20260924 00:24:43.130455 18909 log.cc:824] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:43.130823 18910 tablet_bootstrap.cc:654] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:43.130928 18911 tablet_bootstrap.cc:654] T 530be28e16ee4f899302b84dda4436aa P 75f71cbdbd9f451db188a60292d52bf8: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:43.131374 18909 tablet_bootstrap.cc:492] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602: No bootstrap required, opened a new log
I20260924 00:24:43.131413 18910 log.cc:824] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:43.131422 18911 log.cc:824] T 530be28e16ee4f899302b84dda4436aa P 75f71cbdbd9f451db188a60292d52bf8: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:43.131533 18909 ts_tablet_manager.cc:1397] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:24:43.132314 18910 tablet_bootstrap.cc:492] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c: No bootstrap required, opened a new log
I20260924 00:24:43.132395 18910 ts_tablet_manager.cc:1397] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 00:24:43.133798 18909 raft_consensus.cc:348] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7aeebd0e01f424b9bdc66b39bac7602" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43979 } } peers { permanent_uuid: "e81368c22024482b9256b728cca3b62c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 } } peers { permanent_uuid: "75f71cbdbd9f451db188a60292d52bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36745 } }
I20260924 00:24:43.134007 18909 raft_consensus.cc:374] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:43.134047 18909 raft_consensus.cc:729] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7aeebd0e01f424b9bdc66b39bac7602, State: Initialized, Role: FOLLOWER
I20260924 00:24:43.134184 18909 consensus_queue.cc:260] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7aeebd0e01f424b9bdc66b39bac7602" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43979 } } peers { permanent_uuid: "e81368c22024482b9256b728cca3b62c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 } } peers { permanent_uuid: "75f71cbdbd9f451db188a60292d52bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36745 } }
I20260924 00:24:43.134485 18909 ts_tablet_manager.cc:1428] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:24:43.134513 18892 heartbeater.cc:503] Master 127.0.98.62:43521 was elected leader, sending a full tablet report...
I20260924 00:24:43.134406 18910 raft_consensus.cc:348] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7aeebd0e01f424b9bdc66b39bac7602" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43979 } } peers { permanent_uuid: "e81368c22024482b9256b728cca3b62c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 } } peers { permanent_uuid: "75f71cbdbd9f451db188a60292d52bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36745 } }
I20260924 00:24:43.134577 18910 raft_consensus.cc:374] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:43.134624 18910 raft_consensus.cc:729] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e81368c22024482b9256b728cca3b62c, State: Initialized, Role: FOLLOWER
I20260924 00:24:43.134750 18910 consensus_queue.cc:260] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7aeebd0e01f424b9bdc66b39bac7602" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43979 } } peers { permanent_uuid: "e81368c22024482b9256b728cca3b62c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 } } peers { permanent_uuid: "75f71cbdbd9f451db188a60292d52bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36745 } }
I20260924 00:24:43.135039 18910 ts_tablet_manager.cc:1428] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:24:43.135051 18758 heartbeater.cc:503] Master 127.0.98.62:43521 was elected leader, sending a full tablet report...
I20260924 00:24:43.135620 18911 tablet_bootstrap.cc:492] T 530be28e16ee4f899302b84dda4436aa P 75f71cbdbd9f451db188a60292d52bf8: No bootstrap required, opened a new log
I20260924 00:24:43.135699 18911 ts_tablet_manager.cc:1397] T 530be28e16ee4f899302b84dda4436aa P 75f71cbdbd9f451db188a60292d52bf8: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260924 00:24:43.137290 18911 raft_consensus.cc:348] T 530be28e16ee4f899302b84dda4436aa P 75f71cbdbd9f451db188a60292d52bf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7aeebd0e01f424b9bdc66b39bac7602" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43979 } } peers { permanent_uuid: "e81368c22024482b9256b728cca3b62c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 } } peers { permanent_uuid: "75f71cbdbd9f451db188a60292d52bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36745 } }
I20260924 00:24:43.137432 18911 raft_consensus.cc:374] T 530be28e16ee4f899302b84dda4436aa P 75f71cbdbd9f451db188a60292d52bf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:43.137455 18911 raft_consensus.cc:729] T 530be28e16ee4f899302b84dda4436aa P 75f71cbdbd9f451db188a60292d52bf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f71cbdbd9f451db188a60292d52bf8, State: Initialized, Role: FOLLOWER
I20260924 00:24:43.137568 18911 consensus_queue.cc:260] T 530be28e16ee4f899302b84dda4436aa P 75f71cbdbd9f451db188a60292d52bf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7aeebd0e01f424b9bdc66b39bac7602" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43979 } } peers { permanent_uuid: "e81368c22024482b9256b728cca3b62c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 } } peers { permanent_uuid: "75f71cbdbd9f451db188a60292d52bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36745 } }
I20260924 00:24:43.137842 18911 ts_tablet_manager.cc:1428] T 530be28e16ee4f899302b84dda4436aa P 75f71cbdbd9f451db188a60292d52bf8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:24:43.137859 18624 heartbeater.cc:503] Master 127.0.98.62:43521 was elected leader, sending a full tablet report...
W20260924 00:24:43.175182 18759 tablet.cc:2369] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:24:43.225929 18915 raft_consensus.cc:482] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:24:43.226078 18915 raft_consensus.cc:504] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7aeebd0e01f424b9bdc66b39bac7602" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43979 } } peers { permanent_uuid: "e81368c22024482b9256b728cca3b62c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 } } peers { permanent_uuid: "75f71cbdbd9f451db188a60292d52bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36745 } }
I20260924 00:24:43.226513 18915 leader_election.cc:288] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e81368c22024482b9256b728cca3b62c (127.0.98.2:33927), 75f71cbdbd9f451db188a60292d52bf8 (127.0.98.1:36745)
I20260924 00:24:43.229628 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.226789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42657 (local address 127.0.98.1:36745)
0924 00:24:43.226840 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:43.226842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:43.226854 (+ 12us) server_negotiation.cc:407] Connection header received
0924 00:24:43.226887 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:43.226888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:43.226900 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:43.226976 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:43.227141 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.227612 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.228798 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.229242 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.229349 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.229361 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:43.229519 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 00:24:43.229556 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 00:24:43.229628 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.226682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36745 (local address 127.0.98.3:42657)
0924 00:24:43.226761 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:43.226773 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:43.226827 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:43.226982 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:43.226982 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:43.226984 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:43.227060 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.227061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.227706 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.227707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.228687 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.228690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.229350 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.229351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.229444 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.229447 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:24:43.229513 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 00:24:43.229555 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 00:24:43.229728 18907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.226751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36165 (local address 127.0.98.2:33927)
0924 00:24:43.226870 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:43.226873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:43.227002 (+ 129us) server_negotiation.cc:407] Connection header received
0924 00:24:43.227086 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:43.227087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:43.227105 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:43.227163 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:43.227324 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.227828 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.228893 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.229336 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.229438 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.229448 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:43.229598 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 00:24:43.229632 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 00:24:43.229786 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.226684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33927 (local address 127.0.98.3:36165)
0924 00:24:43.226928 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:43.226937 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:43.227016 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:43.227166 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:43.227167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:43.227169 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:43.227249 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.227250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.227922 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.227923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.228779 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.228781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.229443 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.229444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.229518 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.229521 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:24:43.229580 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 00:24:43.229622 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":14,"thread_start_us":115,"threads_started":1}
I20260924 00:24:43.230038 18578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "530be28e16ee4f899302b84dda4436aa" candidate_uuid: "b7aeebd0e01f424b9bdc66b39bac7602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f71cbdbd9f451db188a60292d52bf8" is_pre_election: true
I20260924 00:24:43.230134 18917 raft_consensus.cc:482] T 530be28e16ee4f899302b84dda4436aa P 75f71cbdbd9f451db188a60292d52bf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:24:43.230206 18917 raft_consensus.cc:504] T 530be28e16ee4f899302b84dda4436aa P 75f71cbdbd9f451db188a60292d52bf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7aeebd0e01f424b9bdc66b39bac7602" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43979 } } peers { permanent_uuid: "e81368c22024482b9256b728cca3b62c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 } } peers { permanent_uuid: "75f71cbdbd9f451db188a60292d52bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36745 } }
I20260924 00:24:43.230346 18578 raft_consensus.cc:2454] T 530be28e16ee4f899302b84dda4436aa P 75f71cbdbd9f451db188a60292d52bf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7aeebd0e01f424b9bdc66b39bac7602 in term 0.
I20260924 00:24:43.230476 18712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "530be28e16ee4f899302b84dda4436aa" candidate_uuid: "b7aeebd0e01f424b9bdc66b39bac7602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e81368c22024482b9256b728cca3b62c" is_pre_election: true
I20260924 00:24:43.230530 18917 leader_election.cc:288] T 530be28e16ee4f899302b84dda4436aa P 75f71cbdbd9f451db188a60292d52bf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7aeebd0e01f424b9bdc66b39bac7602 (127.0.98.3:43979), e81368c22024482b9256b728cca3b62c (127.0.98.2:33927)
I20260924 00:24:43.230635 18712 raft_consensus.cc:2454] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7aeebd0e01f424b9bdc66b39bac7602 in term 0.
I20260924 00:24:43.230899 18779 leader_election.cc:302] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7aeebd0e01f424b9bdc66b39bac7602, e81368c22024482b9256b728cca3b62c; no voters:
I20260924 00:24:43.231048 18915 raft_consensus.cc:2781] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:24:43.231099 18915 raft_consensus.cc:482] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:24:43.231130 18915 raft_consensus.cc:3037] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:43.232110 18915 raft_consensus.cc:504] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7aeebd0e01f424b9bdc66b39bac7602" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43979 } } peers { permanent_uuid: "e81368c22024482b9256b728cca3b62c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 } } peers { permanent_uuid: "75f71cbdbd9f451db188a60292d52bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36745 } }
I20260924 00:24:43.232280 18915 leader_election.cc:288] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602 [CANDIDATE]: Term 1 election: Requested vote from peers e81368c22024482b9256b728cca3b62c (127.0.98.2:33927), 75f71cbdbd9f451db188a60292d52bf8 (127.0.98.1:36745)
I20260924 00:24:43.232461 18712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "530be28e16ee4f899302b84dda4436aa" candidate_uuid: "b7aeebd0e01f424b9bdc66b39bac7602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e81368c22024482b9256b728cca3b62c"
I20260924 00:24:43.232571 18712 raft_consensus.cc:3037] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:43.233460 18712 raft_consensus.cc:2454] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7aeebd0e01f424b9bdc66b39bac7602 in term 1.
I20260924 00:24:43.233655 18779 leader_election.cc:302] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7aeebd0e01f424b9bdc66b39bac7602, e81368c22024482b9256b728cca3b62c; no voters:
I20260924 00:24:43.233759 18578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "530be28e16ee4f899302b84dda4436aa" candidate_uuid: "b7aeebd0e01f424b9bdc66b39bac7602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75f71cbdbd9f451db188a60292d52bf8"
I20260924 00:24:43.233834 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.230695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54399 (local address 127.0.98.3:43979)
0924 00:24:43.230734 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:43.230736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:43.230744 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:24:43.230832 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:43.230833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:43.230841 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:43.230924 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:43.231069 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.231744 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.233010 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.233447 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.233544 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.233554 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:43.233742 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 00:24:43.233783 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:24:43.233845 18626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.230666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43979 (local address 127.0.98.1:54399)
0924 00:24:43.230703 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:43.230713 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:43.230753 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:43.230928 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:43.230928 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:43.230930 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:43.230988 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.230990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.231821 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.231822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.232918 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.232920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.233579 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.233580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.233674 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.233678 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:24:43.233736 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 00:24:43.233775 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 00:24:43.233842 18578 raft_consensus.cc:3037] T 530be28e16ee4f899302b84dda4436aa P 75f71cbdbd9f451db188a60292d52bf8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:43.233961 18915 raft_consensus.cc:2781] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:43.234107 18915 raft_consensus.cc:686] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602 [term 1 LEADER]: Becoming Leader. State: Replica: b7aeebd0e01f424b9bdc66b39bac7602, State: Running, Role: LEADER
I20260924 00:24:43.234148 18846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "530be28e16ee4f899302b84dda4436aa" candidate_uuid: "75f71cbdbd9f451db188a60292d52bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7aeebd0e01f424b9bdc66b39bac7602" is_pre_election: true
I20260924 00:24:43.234200 18915 consensus_queue.cc:237] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7aeebd0e01f424b9bdc66b39bac7602" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43979 } } peers { permanent_uuid: "e81368c22024482b9256b728cca3b62c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 } } peers { permanent_uuid: "75f71cbdbd9f451db188a60292d52bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36745 } }
I20260924 00:24:43.234812 18578 raft_consensus.cc:2454] T 530be28e16ee4f899302b84dda4436aa P 75f71cbdbd9f451db188a60292d52bf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7aeebd0e01f424b9bdc66b39bac7602 in term 1.
I20260924 00:24:43.234992 18432 catalog_manager.cc:5495] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602 reported cstate change: term changed from 0 to 1, leader changed from <none> to b7aeebd0e01f424b9bdc66b39bac7602 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b7aeebd0e01f424b9bdc66b39bac7602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7aeebd0e01f424b9bdc66b39bac7602" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e81368c22024482b9256b728cca3b62c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75f71cbdbd9f451db188a60292d52bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36745 } health_report { overall_health: UNKNOWN } } }
I20260924 00:24:43.235189 18907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.230979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38883 (local address 127.0.98.2:33927)
0924 00:24:43.231235 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:43.231236 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:43.231290 (+ 54us) server_negotiation.cc:407] Connection header received
0924 00:24:43.231353 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:43.231354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:43.231363 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:43.231417 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:43.231551 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.232140 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.233686 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.234217 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.234294 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.234303 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:43.235092 (+ 789us) server_negotiation.cc:299] Negotiation successful
0924 00:24:43.235130 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235}
I20260924 00:24:43.235203 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.230670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33927 (local address 127.0.98.1:38883)
0924 00:24:43.231230 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:43.231241 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:43.231304 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:43.231421 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:43.231421 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:43.231423 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:43.231487 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.231488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.232276 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.232277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.233514 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.233517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.234934 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.234936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.235004 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.235009 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:24:43.235075 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 00:24:43.235113 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":330,"thread_start_us":91,"threads_started":1}
I20260924 00:24:43.235394 18712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "530be28e16ee4f899302b84dda4436aa" candidate_uuid: "75f71cbdbd9f451db188a60292d52bf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e81368c22024482b9256b728cca3b62c" is_pre_election: true
I20260924 00:24:43.235495 18712 raft_consensus.cc:2379] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 75f71cbdbd9f451db188a60292d52bf8 in current term 1: Already voted for candidate b7aeebd0e01f424b9bdc66b39bac7602 in this term.
I20260924 00:24:43.235890 18511 leader_election.cc:302] T 530be28e16ee4f899302b84dda4436aa P 75f71cbdbd9f451db188a60292d52bf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 75f71cbdbd9f451db188a60292d52bf8; no voters: b7aeebd0e01f424b9bdc66b39bac7602, e81368c22024482b9256b728cca3b62c
I20260924 00:24:43.236130 18917 raft_consensus.cc:2726] T 530be28e16ee4f899302b84dda4436aa P 75f71cbdbd9f451db188a60292d52bf8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260924 00:24:43.252105 18625 tablet.cc:2369] T 530be28e16ee4f899302b84dda4436aa P 75f71cbdbd9f451db188a60292d52bf8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:24:43.270445 18908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.267190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.0.98.3:43979)
0924 00:24:43.267250 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:43.267252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:43.267261 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:24:43.267332 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:43.267333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:43.267361 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:43.267449 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:24:43.267607 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.268319 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.269520 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.269708 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.269768 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.270044 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:24:43.270153 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:24:43.270217 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:43.270286 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 00:24:43.270307 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 00:24:43.272519 18712 raft_consensus.cc:1261] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c [term 1 FOLLOWER]: Refusing update from remote peer b7aeebd0e01f424b9bdc66b39bac7602: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:24:43.272539 18578 raft_consensus.cc:1261] T 530be28e16ee4f899302b84dda4436aa P 75f71cbdbd9f451db188a60292d52bf8 [term 1 FOLLOWER]: Refusing update from remote peer b7aeebd0e01f424b9bdc66b39bac7602: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:24:43.272850 18915 consensus_queue.cc:1035] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e81368c22024482b9256b728cca3b62c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:24:43.273005 18920 consensus_queue.cc:1035] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75f71cbdbd9f451db188a60292d52bf8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:24:43.277714 18432 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:24:43.277824 18432 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:24:43.278599 18432 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e4ca6d4098b41849dc803e9cd80c64d: Sending DeleteTablet for 3 replicas of tablet 530be28e16ee4f899302b84dda4436aa
I20260924 00:24:43.278939 18692 tablet_service.cc:1514] Processing DeleteTablet for tablet 530be28e16ee4f899302b84dda4436aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:24:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33898
I20260924 00:24:43.278946 18826 tablet_service.cc:1514] Processing DeleteTablet for tablet 530be28e16ee4f899302b84dda4436aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:24:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49118
I20260924 00:24:43.279207 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 18495
I20260924 00:24:43.279335 18558 tablet_service.cc:1514] Processing DeleteTablet for tablet 530be28e16ee4f899302b84dda4436aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:24:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36512
I20260924 00:24:43.280470 18934 tablet_replica.cc:330] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c: stopping tablet replica
I20260924 00:24:43.280587 18934 raft_consensus.cc:2229] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:24:43.280699 18934 raft_consensus.cc:2258] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:24:43.279290 18933 tablet_replica.cc:330] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602: stopping tablet replica
I20260924 00:24:43.280862 18933 raft_consensus.cc:2229] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602 [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:24:43.280977 18933 raft_consensus.cc:2258] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:24:43.281510 18934 ts_tablet_manager.cc:1905] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:24:43.281545 18933 ts_tablet_manager.cc:1905] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:24:43.283337 18933 ts_tablet_manager.cc:1918] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:24:43.283510 18933 log.cc:1196] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/530be28e16ee4f899302b84dda4436aa
I20260924 00:24:43.283586 18934 ts_tablet_manager.cc:1918] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:24:43.283746 18934 log.cc:1196] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790209154318147-392-0/minicluster-data/ts-1/wal/wals/530be28e16ee4f899302b84dda4436aa
I20260924 00:24:43.283859 18933 ts_tablet_manager.cc:1939] T 530be28e16ee4f899302b84dda4436aa P b7aeebd0e01f424b9bdc66b39bac7602: Deleting consensus metadata
I20260924 00:24:43.284086 18934 ts_tablet_manager.cc:1939] T 530be28e16ee4f899302b84dda4436aa P e81368c22024482b9256b728cca3b62c: Deleting consensus metadata
I20260924 00:24:43.284375 18418 catalog_manager.cc:4841] TS b7aeebd0e01f424b9bdc66b39bac7602 (127.0.98.3:43979): tablet 530be28e16ee4f899302b84dda4436aa (table test-table [id=98327fe20d7b4fd0ad53a27ffe86c3a3]) successfully deleted
I20260924 00:24:43.284794 18417 catalog_manager.cc:4841] TS e81368c22024482b9256b728cca3b62c (127.0.98.2:33927): tablet 530be28e16ee4f899302b84dda4436aa (table test-table [id=98327fe20d7b4fd0ad53a27ffe86c3a3]) successfully deleted
W20260924 00:24:43.286098 18419 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36745 (error 108)
I20260924 00:24:43.286336 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 18629
I20260924 00:24:43.287072 18904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.286461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36745 (local address 127.0.0.1:36514)
0924 00:24:43.287010 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:43.287043 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531}
W20260924 00:24:43.287195 18419 catalog_manager.cc:4584] TS 75f71cbdbd9f451db188a60292d52bf8 (127.0.98.1:36745): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 530be28e16ee4f899302b84dda4436aa: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36745: connect: Connection refused (error 111)
I20260924 00:24:43.291990 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 18763
I20260924 00:24:43.296808 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 18401
2026-09-24T00:24:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2073 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T00:24:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:24:43Z Disabled control of system clock
I20260924 00:24:43.324306 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46545
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 00:24:43.406185 18939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:43.406430 18939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:43.406452 18939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:43.407680 18939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:24:43.407737 18939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:43.407749 18939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:43.407763 18939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:24:43.407775 18939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:24:43.409260 18939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46545
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:43.409485 18939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:43.409737 18939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:43.415079 18944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:43.415067 18947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:43.415067 18945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:43.415650 18939 server_base.cc:1034] running on GCE node
I20260924 00:24:43.415891 18939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:43.416234 18939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:43.417423 18939 hybrid_clock.cc:648] HybridClock initialized: now 1790209483417401 us; error 32 us; skew 500 ppm
I20260924 00:24:43.418807 18939 webserver.cc:460] Webserver started at http://127.0.98.62:43895/ using document root <none> and password file <none>
I20260924 00:24:43.419070 18939 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:43.419140 18939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:43.419291 18939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:43.420475 18939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "6d1187a11d7741cc8539987b38812dae"
format_stamp: "Formatted at 2026-09-24 00:24:43 on dist-test-slave-777x"
server_key: "0815576e85803c479c5c8baf1ac589a7"
server_key_iv: "8f239b7cdfcbd7f64220872ee5dbdcbb"
server_key_version: "encryptionkey@0"
I20260924 00:24:43.420846 18939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "6d1187a11d7741cc8539987b38812dae"
format_stamp: "Formatted at 2026-09-24 00:24:43 on dist-test-slave-777x"
server_key: "0815576e85803c479c5c8baf1ac589a7"
server_key_iv: "8f239b7cdfcbd7f64220872ee5dbdcbb"
server_key_version: "encryptionkey@0"
I20260924 00:24:43.422187 18939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 00:24:43.423053 18953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:43.423242 18939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:43.423341 18939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "6d1187a11d7741cc8539987b38812dae"
format_stamp: "Formatted at 2026-09-24 00:24:43 on dist-test-slave-777x"
server_key: "0815576e85803c479c5c8baf1ac589a7"
server_key_iv: "8f239b7cdfcbd7f64220872ee5dbdcbb"
server_key_version: "encryptionkey@0"
I20260924 00:24:43.423427 18939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:43.440495 18939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:43.440843 18939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:43.440999 18939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:43.445068 18939 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46545
I20260924 00:24:43.445116 19005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46545 every 8 connection(s)
I20260924 00:24:43.445488 18939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:24:43.446512 19006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:43.448827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 18939
I20260924 00:24:43.448959 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:24:43.449211 392 external_mini_cluster.cc:1428] Setting key 223f7d44afaa166db676a18530efa38d
I20260924 00:24:43.449389 19006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae: Bootstrap starting.
I20260924 00:24:43.450215 19006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:43.450557 19006 log.cc:824] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:43.451354 19006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae: No bootstrap required, opened a new log
I20260924 00:24:43.453679 19006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1187a11d7741cc8539987b38812dae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46545 } }
I20260924 00:24:43.453909 19006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:43.453958 19006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1187a11d7741cc8539987b38812dae, State: Initialized, Role: FOLLOWER
I20260924 00:24:43.454119 19006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1187a11d7741cc8539987b38812dae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46545 } }
I20260924 00:24:43.454217 19006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:24:43.454245 19006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:24:43.454324 19006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:43.455122 19006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1187a11d7741cc8539987b38812dae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46545 } }
I20260924 00:24:43.455272 19006 leader_election.cc:302] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d1187a11d7741cc8539987b38812dae; no voters:
I20260924 00:24:43.455417 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.449829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58092 (local address 127.0.98.62:46545)
0924 00:24:43.450062 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:43.450070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:43.450097 (+ 27us) server_negotiation.cc:407] Connection header received
0924 00:24:43.450591 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:43.450598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:43.450689 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:43.450874 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:43.451103 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.452087 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.453270 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.453693 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.453793 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.454621 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:43.454625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:43.454625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:43.454842 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:43.454877 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:43.454888 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:43.454943 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:43.455059 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 00:24:43.455123 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":104,"threads_started":1}
I20260924 00:24:43.455540 19006 leader_election.cc:288] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:24:43.455739 19012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:43.455847 19006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:24:43.455911 19012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae [term 1 LEADER]: Becoming Leader. State: Replica: 6d1187a11d7741cc8539987b38812dae, State: Running, Role: LEADER
I20260924 00:24:43.456115 19012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1187a11d7741cc8539987b38812dae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46545 } }
I20260924 00:24:43.456640 19011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d1187a11d7741cc8539987b38812dae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1187a11d7741cc8539987b38812dae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46545 } } }
I20260924 00:24:43.456629 19013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d1187a11d7741cc8539987b38812dae. Latest consensus state: current_term: 1 leader_uuid: "6d1187a11d7741cc8539987b38812dae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1187a11d7741cc8539987b38812dae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46545 } } }
I20260924 00:24:43.456743 19011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:43.456745 19013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:43.456987 19020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:24:43.457552 19020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:24:43.459121 19020 catalog_manager.cc:1347] Generated new cluster ID: 89105281dc4f47e69e07fa7d333a4d95
I20260924 00:24:43.459199 19020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:24:43.465909 19020 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:24:43.466643 19020 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:24:43.469838 19020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae: Generated new TSK 0
I20260924 00:24:43.470033 19020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:24:43.474838 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46545
--builtin_ntp_servers=127.0.98.20:33965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 00:24:43.568107 19030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:43.568415 19030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:43.568441 19030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:43.569691 19030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:43.569739 19030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:43.569767 19030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 00:24:43.571190 19030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:43.571468 19030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:43.571821 19030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:43.578217 19036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:43.578217 19035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:43.578235 19038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:43.578559 19030 server_base.cc:1034] running on GCE node
I20260924 00:24:43.578775 19030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:43.579061 19030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:43.580245 19030 hybrid_clock.cc:648] HybridClock initialized: now 1790209483580225 us; error 31 us; skew 500 ppm
I20260924 00:24:43.581816 19030 webserver.cc:460] Webserver started at http://127.0.98.1:37437/ using document root <none> and password file <none>
I20260924 00:24:43.582166 19030 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:43.582242 19030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:43.582402 19030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:43.583812 19030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-0/data/instance:
uuid: "4790979c758e4af09a8f4e8bbc94a1f3"
format_stamp: "Formatted at 2026-09-24 00:24:43 on dist-test-slave-777x"
server_key: "84c23566b7f33072e3f2ee874d90ac9c"
server_key_iv: "600624f87b3395fcf3570586410f9d36"
server_key_version: "encryptionkey@0"
I20260924 00:24:43.584299 19030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4790979c758e4af09a8f4e8bbc94a1f3"
format_stamp: "Formatted at 2026-09-24 00:24:43 on dist-test-slave-777x"
server_key: "84c23566b7f33072e3f2ee874d90ac9c"
server_key_iv: "600624f87b3395fcf3570586410f9d36"
server_key_version: "encryptionkey@0"
I20260924 00:24:43.585804 19030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:43.586678 19044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:43.586907 19030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:43.587013 19030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-0/wal
uuid: "4790979c758e4af09a8f4e8bbc94a1f3"
format_stamp: "Formatted at 2026-09-24 00:24:43 on dist-test-slave-777x"
server_key: "84c23566b7f33072e3f2ee874d90ac9c"
server_key_iv: "600624f87b3395fcf3570586410f9d36"
server_key_version: "encryptionkey@0"
I20260924 00:24:43.587090 19030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:43.615945 19030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:43.616273 19030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:43.616395 19030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:43.616602 19030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:24:43.616997 19030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:24:43.617038 19030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:43.617092 19030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:24:43.617117 19030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:43.623505 19030 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44989
I20260924 00:24:43.623621 19157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44989 every 8 connection(s)
I20260924 00:24:43.623960 19030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-0/data/info.pb
I20260924 00:24:43.630016 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.624800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56445 (local address 127.0.98.62:46545)
0924 00:24:43.624885 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:43.624887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:43.625158 (+ 271us) server_negotiation.cc:407] Connection header received
0924 00:24:43.625666 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:43.625667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:43.625696 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:43.625753 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:43.626569 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.627337 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.628467 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.628676 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.628749 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.629643 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:43.629645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:43.629645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:43.629650 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:43.629665 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:43.629670 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:43.629803 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:43.629927 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 00:24:43.629950 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 00:24:43.630246 19160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.624770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46545 (local address 127.0.98.1:56445)
0924 00:24:43.625035 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:43.625072 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:43.625573 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:43.625822 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:43.625826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:43.625835 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:24:43.626413 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.626419 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.627457 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.627458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.628384 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.628387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.628756 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.628757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.628870 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.629502 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:24:43.629502 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:24:43.629533 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:24:43.629557 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:24:43.629811 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:24:43.629817 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:24:43.629820 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:24:43.629895 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 00:24:43.629985 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":163,"threads_started":1}
I20260924 00:24:43.630949 19158 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46545
I20260924 00:24:43.631057 19158 heartbeater.cc:464] Registering TS with master...
I20260924 00:24:43.631254 19158 heartbeater.cc:511] Master 127.0.98.62:46545 requested a full tablet report, sending...
I20260924 00:24:43.631793 18970 ts_manager.cc:194] Registered new tserver with Master: 4790979c758e4af09a8f4e8bbc94a1f3 (127.0.98.1:44989)
I20260924 00:24:43.632524 18970 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:56445
I20260924 00:24:43.633893 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 19030
I20260924 00:24:43.633992 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-0/wal/instance
I20260924 00:24:43.634143 392 external_mini_cluster.cc:1428] Setting key aee81f4c9dd91a58c9d8c4ad67ba86b6
I20260924 00:24:43.635027 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46545
--builtin_ntp_servers=127.0.98.20:33965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 00:24:43.720877 19161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:43.721108 19161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:43.721134 19161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:43.722288 19161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:43.722332 19161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:43.722359 19161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 00:24:43.723680 19161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:43.723889 19161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:43.724156 19161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:43.729782 19167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:43.729820 19169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:43.729893 19166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:43.730006 19161 server_base.cc:1034] running on GCE node
I20260924 00:24:43.730259 19161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:43.730557 19161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:43.731734 19161 hybrid_clock.cc:648] HybridClock initialized: now 1790209483731707 us; error 44 us; skew 500 ppm
I20260924 00:24:43.733214 19161 webserver.cc:460] Webserver started at http://127.0.98.2:34841/ using document root <none> and password file <none>
I20260924 00:24:43.733479 19161 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:43.733527 19161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:43.733680 19161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:43.734798 19161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-1/data/instance:
uuid: "2fcfb881df874ff194bde9da058d9dff"
format_stamp: "Formatted at 2026-09-24 00:24:43 on dist-test-slave-777x"
server_key: "584fb0b6bcd0fdc56d65b317cfcf2f22"
server_key_iv: "456f4f9e5a829b8826f57c565276180b"
server_key_version: "encryptionkey@0"
I20260924 00:24:43.735185 19161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2fcfb881df874ff194bde9da058d9dff"
format_stamp: "Formatted at 2026-09-24 00:24:43 on dist-test-slave-777x"
server_key: "584fb0b6bcd0fdc56d65b317cfcf2f22"
server_key_iv: "456f4f9e5a829b8826f57c565276180b"
server_key_version: "encryptionkey@0"
I20260924 00:24:43.736622 19161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 00:24:43.737396 19175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:43.737607 19161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:24:43.737713 19161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-1/wal
uuid: "2fcfb881df874ff194bde9da058d9dff"
format_stamp: "Formatted at 2026-09-24 00:24:43 on dist-test-slave-777x"
server_key: "584fb0b6bcd0fdc56d65b317cfcf2f22"
server_key_iv: "456f4f9e5a829b8826f57c565276180b"
server_key_version: "encryptionkey@0"
I20260924 00:24:43.737854 19161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:43.751648 19161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:43.752002 19161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:43.752161 19161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:43.752472 19161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:24:43.752838 19161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:24:43.752897 19161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:43.752942 19161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:24:43.752972 19161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:43.759565 19161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33239
I20260924 00:24:43.759626 19288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33239 every 8 connection(s)
I20260924 00:24:43.760003 19161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-1/data/info.pb
I20260924 00:24:43.765755 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.760744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51591 (local address 127.0.98.62:46545)
0924 00:24:43.760816 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:43.760818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:43.761069 (+ 251us) server_negotiation.cc:407] Connection header received
0924 00:24:43.761637 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:43.761638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:43.761652 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:43.761724 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:43.762472 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.763052 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.764190 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.764437 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.764530 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.765446 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:43.765448 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:43.765448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:43.765454 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:43.765471 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:43.765476 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:43.765531 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:43.765632 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 00:24:43.765664 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 00:24:43.766023 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.760755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46545 (local address 127.0.98.2:51591)
0924 00:24:43.760938 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:43.760975 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:43.761524 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:43.761807 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:43.761811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:43.761819 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:24:43.762370 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.762374 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.763167 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.763168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.764071 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.764073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.764543 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.764544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.764699 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.765257 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:24:43.765257 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:24:43.765319 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:24:43.765346 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:24:43.765537 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:24:43.765542 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:24:43.765545 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:24:43.765635 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 00:24:43.765746 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260924 00:24:43.766582 19289 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46545
I20260924 00:24:43.766685 19289 heartbeater.cc:464] Registering TS with master...
I20260924 00:24:43.766853 19289 heartbeater.cc:511] Master 127.0.98.62:46545 requested a full tablet report, sending...
I20260924 00:24:43.767205 18970 ts_manager.cc:194] Registered new tserver with Master: 2fcfb881df874ff194bde9da058d9dff (127.0.98.2:33239)
I20260924 00:24:43.767555 18970 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:51591
I20260924 00:24:43.770147 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 19161
I20260924 00:24:43.770255 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-1/wal/instance
I20260924 00:24:43.770416 392 external_mini_cluster.cc:1428] Setting key 72659a9c96fad7ef474f993de5e50508
I20260924 00:24:43.771544 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46545
--builtin_ntp_servers=127.0.98.20:33965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 00:24:43.854936 19292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:43.855170 19292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:43.855194 19292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:43.856611 19292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:43.856657 19292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:43.856683 19292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 00:24:43.858067 19292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:43.858265 19292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:43.858512 19292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:43.863618 19300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:43.863627 19298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:43.863898 19292 server_base.cc:1034] running on GCE node
W20260924 00:24:43.863646 19297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:43.864253 19292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:43.864516 19292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:43.865695 19292 hybrid_clock.cc:648] HybridClock initialized: now 1790209483865688 us; error 37 us; skew 500 ppm
I20260924 00:24:43.867100 19292 webserver.cc:460] Webserver started at http://127.0.98.3:37397/ using document root <none> and password file <none>
I20260924 00:24:43.867352 19292 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:43.867447 19292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:43.867586 19292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:43.868762 19292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-2/data/instance:
uuid: "e230f761c66040eba8399dcbd1ef5cac"
format_stamp: "Formatted at 2026-09-24 00:24:43 on dist-test-slave-777x"
server_key: "eb67f100527a14ca7c6e25ed3c0a2a6c"
server_key_iv: "c5e89dac2fcc85e9c64df99ba8b9c8ef"
server_key_version: "encryptionkey@0"
I20260924 00:24:43.869119 19292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e230f761c66040eba8399dcbd1ef5cac"
format_stamp: "Formatted at 2026-09-24 00:24:43 on dist-test-slave-777x"
server_key: "eb67f100527a14ca7c6e25ed3c0a2a6c"
server_key_iv: "c5e89dac2fcc85e9c64df99ba8b9c8ef"
server_key_version: "encryptionkey@0"
I20260924 00:24:43.870741 19292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 00:24:43.871593 19306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:43.871767 19292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 00:24:43.871869 19292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-2/wal
uuid: "e230f761c66040eba8399dcbd1ef5cac"
format_stamp: "Formatted at 2026-09-24 00:24:43 on dist-test-slave-777x"
server_key: "eb67f100527a14ca7c6e25ed3c0a2a6c"
server_key_iv: "c5e89dac2fcc85e9c64df99ba8b9c8ef"
server_key_version: "encryptionkey@0"
I20260924 00:24:43.871950 19292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:43.883888 19292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:43.884253 19292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:43.884411 19292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:43.884639 19292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 00:24:43.884974 19292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 00:24:43.885025 19292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:43.885061 19292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 00:24:43.885087 19292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:43.890878 19292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43793
I20260924 00:24:43.890998 19419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43793 every 8 connection(s)
I20260924 00:24:43.891281 19292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-2/data/info.pb
I20260924 00:24:43.896749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 19292
I20260924 00:24:43.896888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-2/wal/instance
I20260924 00:24:43.897127 392 external_mini_cluster.cc:1428] Setting key c14ddb2a78503ee056440fc716200046
I20260924 00:24:43.897239 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.892256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32837 (local address 127.0.98.62:46545)
0924 00:24:43.892315 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:43.892317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:43.892569 (+ 252us) server_negotiation.cc:407] Connection header received
0924 00:24:43.893185 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:43.893187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:43.893199 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:43.893272 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:43.894010 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.894595 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.895659 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.895871 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.895936 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.896904 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:43.896906 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:43.896906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:43.896911 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:43.896931 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:43.896945 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:43.897005 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:43.897098 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 00:24:43.897124 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 00:24:43.897454 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.892211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46545 (local address 127.0.98.3:32837)
0924 00:24:43.892449 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:43.892485 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:43.893068 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:43.893331 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:43.893335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:43.893343 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:24:43.893892 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.893897 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.894691 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.894692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.895564 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.895567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.895956 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.895957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.896077 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.896693 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 00:24:43.896694 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:24:43.896740 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 00:24:43.896773 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:24:43.897009 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:24:43.897016 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:24:43.897019 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:24:43.897095 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 00:24:43.897212 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":129,"threads_started":1}
I20260924 00:24:43.898061 19420 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46545
I20260924 00:24:43.898171 19420 heartbeater.cc:464] Registering TS with master...
I20260924 00:24:43.898388 19420 heartbeater.cc:511] Master 127.0.98.62:46545 requested a full tablet report, sending...
I20260924 00:24:43.898741 18970 ts_manager.cc:194] Registered new tserver with Master: e230f761c66040eba8399dcbd1ef5cac (127.0.98.3:43793)
I20260924 00:24:43.899153 18970 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:32837
I20260924 00:24:43.909278 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 00:24:43.914274 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.910963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58108 (local address 127.0.98.62:46545)
0924 00:24:43.911047 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:43.911055 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:43.911115 (+ 60us) server_negotiation.cc:407] Connection header received
0924 00:24:43.911175 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:43.911176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:43.911185 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:43.911314 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:43.911487 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.912029 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.913080 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.913286 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.913360 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.914044 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:43.914046 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:43.914046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:43.914051 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:43.914067 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:43.914071 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:43.914130 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:43.914179 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 00:24:43.914207 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 00:24:43.916131 18970 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 00:24:43.917984 18970 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 00:24:43.925005 19430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.920607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36574 (local address 127.0.98.3:43793)
0924 00:24:43.920887 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:43.920892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:43.920906 (+ 14us) server_negotiation.cc:407] Connection header received
0924 00:24:43.920955 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:43.920957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:43.920976 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:43.921096 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:43.921421 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.922553 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.923778 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.924547 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.924625 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.924726 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:43.924851 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 00:24:43.924891 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":92,"threads_started":1}
I20260924 00:24:43.925012 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.920523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43793 (local address 127.0.0.1:36574)
0924 00:24:43.920658 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:43.920701 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:43.920764 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:43.921139 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:43.921141 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:43.921143 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:43.921219 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.921221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.922651 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.922652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.923687 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.923689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.924635 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.924636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.924716 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.924732 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 00:24:43.924832 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 00:24:43.924890 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260924 00:24:43.925199 19433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.920956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34934 (local address 127.0.98.1:44989)
0924 00:24:43.921182 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:43.921187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:43.921198 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:24:43.921241 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:43.921244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:43.921261 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:43.921377 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:43.921568 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.922332 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.923929 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.924676 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.924737 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.924810 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:43.925058 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 00:24:43.925099 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":100,"threads_started":1}
I20260924 00:24:43.925222 19429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.920524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33239 (local address 127.0.0.1:43830)
0924 00:24:43.920908 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:43.920920 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:43.920976 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:43.921508 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:43.921509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:43.921511 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:43.921557 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.921559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.922784 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.922785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.923989 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.923991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.924990 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.924991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.925060 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.925065 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:24:43.925102 (+ 37us) client_negotiation.cc:240] Negotiation successful
0924 00:24:43.925145 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":2,"thread_start_us":109,"threads_started":1}
I20260924 00:24:43.925398 19432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.920755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44989 (local address 127.0.0.1:34934)
0924 00:24:43.920962 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:43.920973 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:43.921023 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:43.921455 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:43.921456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:43.921459 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:43.921504 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.921505 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.922425 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.922426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.923823 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:43.923826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.924847 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.924848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:43.924917 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.924921 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 00:24:43.924974 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 00:24:43.925014 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":4,"thread_start_us":68,"threads_started":1}
I20260924 00:24:43.925804 19431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:43.920728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43830 (local address 127.0.98.2:33239)
0924 00:24:43.921226 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:43.921231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:43.921240 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:24:43.921278 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:43.921280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:43.921295 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:43.921408 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:43.921643 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.922692 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.924079 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:43.924889 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:43.925358 (+ 469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:43.925463 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:43.925663 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 00:24:43.925715 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":76,"threads_started":1}
I20260924 00:24:43.926234 19092 tablet_service.cc:1467] Processing CreateTablet for tablet 01f8d19232d14e18b8181b4577725b0e (DEFAULT_TABLE table=test-table [id=97b75aaa8aae470980ebd340b0d3d6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:24:43.926339 19354 tablet_service.cc:1467] Processing CreateTablet for tablet 01f8d19232d14e18b8181b4577725b0e (DEFAULT_TABLE table=test-table [id=97b75aaa8aae470980ebd340b0d3d6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:24:43.926403 19223 tablet_service.cc:1467] Processing CreateTablet for tablet 01f8d19232d14e18b8181b4577725b0e (DEFAULT_TABLE table=test-table [id=97b75aaa8aae470980ebd340b0d3d6cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 00:24:43.926625 19092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f8d19232d14e18b8181b4577725b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:43.926617 19354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f8d19232d14e18b8181b4577725b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:43.926703 19223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01f8d19232d14e18b8181b4577725b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:43.929453 19434 tablet_bootstrap.cc:492] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac: Bootstrap starting.
I20260924 00:24:43.930289 19434 tablet_bootstrap.cc:654] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:43.930547 19434 log.cc:824] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:43.930835 19436 tablet_bootstrap.cc:492] T 01f8d19232d14e18b8181b4577725b0e P 2fcfb881df874ff194bde9da058d9dff: Bootstrap starting.
I20260924 00:24:43.931262 19434 tablet_bootstrap.cc:492] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac: No bootstrap required, opened a new log
I20260924 00:24:43.931320 19434 ts_tablet_manager.cc:1397] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:24:43.931488 19436 tablet_bootstrap.cc:654] T 01f8d19232d14e18b8181b4577725b0e P 2fcfb881df874ff194bde9da058d9dff: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:43.931789 19436 log.cc:824] T 01f8d19232d14e18b8181b4577725b0e P 2fcfb881df874ff194bde9da058d9dff: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:43.932197 19435 tablet_bootstrap.cc:492] T 01f8d19232d14e18b8181b4577725b0e P 4790979c758e4af09a8f4e8bbc94a1f3: Bootstrap starting.
I20260924 00:24:43.932971 19435 tablet_bootstrap.cc:654] T 01f8d19232d14e18b8181b4577725b0e P 4790979c758e4af09a8f4e8bbc94a1f3: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:43.933122 19436 tablet_bootstrap.cc:492] T 01f8d19232d14e18b8181b4577725b0e P 2fcfb881df874ff194bde9da058d9dff: No bootstrap required, opened a new log
I20260924 00:24:43.933207 19436 ts_tablet_manager.cc:1397] T 01f8d19232d14e18b8181b4577725b0e P 2fcfb881df874ff194bde9da058d9dff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 00:24:43.933347 19435 log.cc:824] T 01f8d19232d14e18b8181b4577725b0e P 4790979c758e4af09a8f4e8bbc94a1f3: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:43.933615 19434 raft_consensus.cc:348] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcfb881df874ff194bde9da058d9dff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33239 } } peers { permanent_uuid: "e230f761c66040eba8399dcbd1ef5cac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43793 } } peers { permanent_uuid: "4790979c758e4af09a8f4e8bbc94a1f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44989 } }
I20260924 00:24:43.933837 19434 raft_consensus.cc:374] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:43.933918 19434 raft_consensus.cc:729] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e230f761c66040eba8399dcbd1ef5cac, State: Initialized, Role: FOLLOWER
I20260924 00:24:43.934049 19434 consensus_queue.cc:260] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcfb881df874ff194bde9da058d9dff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33239 } } peers { permanent_uuid: "e230f761c66040eba8399dcbd1ef5cac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43793 } } peers { permanent_uuid: "4790979c758e4af09a8f4e8bbc94a1f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44989 } }
I20260924 00:24:43.934221 19435 tablet_bootstrap.cc:492] T 01f8d19232d14e18b8181b4577725b0e P 4790979c758e4af09a8f4e8bbc94a1f3: No bootstrap required, opened a new log
I20260924 00:24:43.934307 19435 ts_tablet_manager.cc:1397] T 01f8d19232d14e18b8181b4577725b0e P 4790979c758e4af09a8f4e8bbc94a1f3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 00:24:43.934319 19434 ts_tablet_manager.cc:1428] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:24:43.934438 19420 heartbeater.cc:503] Master 127.0.98.62:46545 was elected leader, sending a full tablet report...
I20260924 00:24:43.936378 19436 raft_consensus.cc:348] T 01f8d19232d14e18b8181b4577725b0e P 2fcfb881df874ff194bde9da058d9dff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcfb881df874ff194bde9da058d9dff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33239 } } peers { permanent_uuid: "e230f761c66040eba8399dcbd1ef5cac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43793 } } peers { permanent_uuid: "4790979c758e4af09a8f4e8bbc94a1f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44989 } }
I20260924 00:24:43.936568 19436 raft_consensus.cc:374] T 01f8d19232d14e18b8181b4577725b0e P 2fcfb881df874ff194bde9da058d9dff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:43.936606 19436 raft_consensus.cc:729] T 01f8d19232d14e18b8181b4577725b0e P 2fcfb881df874ff194bde9da058d9dff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fcfb881df874ff194bde9da058d9dff, State: Initialized, Role: FOLLOWER
I20260924 00:24:43.936736 19436 consensus_queue.cc:260] T 01f8d19232d14e18b8181b4577725b0e P 2fcfb881df874ff194bde9da058d9dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcfb881df874ff194bde9da058d9dff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33239 } } peers { permanent_uuid: "e230f761c66040eba8399dcbd1ef5cac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43793 } } peers { permanent_uuid: "4790979c758e4af09a8f4e8bbc94a1f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44989 } }
I20260924 00:24:43.937026 19436 ts_tablet_manager.cc:1428] T 01f8d19232d14e18b8181b4577725b0e P 2fcfb881df874ff194bde9da058d9dff: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 00:24:43.937078 19289 heartbeater.cc:503] Master 127.0.98.62:46545 was elected leader, sending a full tablet report...
I20260924 00:24:43.937150 19435 raft_consensus.cc:348] T 01f8d19232d14e18b8181b4577725b0e P 4790979c758e4af09a8f4e8bbc94a1f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcfb881df874ff194bde9da058d9dff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33239 } } peers { permanent_uuid: "e230f761c66040eba8399dcbd1ef5cac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43793 } } peers { permanent_uuid: "4790979c758e4af09a8f4e8bbc94a1f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44989 } }
I20260924 00:24:43.937335 19435 raft_consensus.cc:374] T 01f8d19232d14e18b8181b4577725b0e P 4790979c758e4af09a8f4e8bbc94a1f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:43.937389 19435 raft_consensus.cc:729] T 01f8d19232d14e18b8181b4577725b0e P 4790979c758e4af09a8f4e8bbc94a1f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4790979c758e4af09a8f4e8bbc94a1f3, State: Initialized, Role: FOLLOWER
I20260924 00:24:43.937528 19435 consensus_queue.cc:260] T 01f8d19232d14e18b8181b4577725b0e P 4790979c758e4af09a8f4e8bbc94a1f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcfb881df874ff194bde9da058d9dff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33239 } } peers { permanent_uuid: "e230f761c66040eba8399dcbd1ef5cac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43793 } } peers { permanent_uuid: "4790979c758e4af09a8f4e8bbc94a1f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44989 } }
I20260924 00:24:43.937813 19435 ts_tablet_manager.cc:1428] T 01f8d19232d14e18b8181b4577725b0e P 4790979c758e4af09a8f4e8bbc94a1f3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 00:24:43.937851 19158 heartbeater.cc:503] Master 127.0.98.62:46545 was elected leader, sending a full tablet report...
W20260924 00:24:44.010612 19290 tablet.cc:2369] T 01f8d19232d14e18b8181b4577725b0e P 2fcfb881df874ff194bde9da058d9dff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 00:24:44.088061 19440 raft_consensus.cc:482] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 00:24:44.088253 19440 raft_consensus.cc:504] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcfb881df874ff194bde9da058d9dff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33239 } } peers { permanent_uuid: "e230f761c66040eba8399dcbd1ef5cac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43793 } } peers { permanent_uuid: "4790979c758e4af09a8f4e8bbc94a1f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44989 } }
I20260924 00:24:44.088790 19440 leader_election.cc:288] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fcfb881df874ff194bde9da058d9dff (127.0.98.2:33239), 4790979c758e4af09a8f4e8bbc94a1f3 (127.0.98.1:44989)
I20260924 00:24:44.092948 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:44.089015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33239 (local address 127.0.98.3:36175)
0924 00:24:44.089070 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:44.089083 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:44.089132 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:44.089342 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:44.089343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:44.089345 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:44.089419 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:44.089421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:44.090379 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:44.090381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:44.091727 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:44.091730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:44.092627 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:44.092628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:44.092719 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:44.092724 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 00:24:44.092795 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 00:24:44.092844 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":56}
I20260924 00:24:44.092953 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:44.089155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44989 (local address 127.0.98.3:39841)
0924 00:24:44.089318 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:44.089328 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:24:44.089379 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:24:44.089552 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:24:44.089552 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:24:44.089554 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 00:24:44.089601 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:44.089603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:44.090440 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:44.090440 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:44.091727 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:24:44.091730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:24:44.092633 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:24:44.092633 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:24:44.092721 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:44.092724 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 00:24:44.092792 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 00:24:44.092841 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":7,"thread_start_us":117,"threads_started":1}
I20260924 00:24:44.093063 19433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:44.089230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39841 (local address 127.0.98.1:44989)
0924 00:24:44.089302 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:44.089304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:44.089373 (+ 69us) server_negotiation.cc:407] Connection header received
0924 00:24:44.089449 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:44.089449 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:44.089476 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:44.089536 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:44.089667 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:44.090369 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:44.091854 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:44.092522 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:44.092616 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:44.092629 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:44.092937 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 00:24:44.092986 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 00:24:44.093063 19431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:44.089125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36175 (local address 127.0.98.2:33239)
0924 00:24:44.089191 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:44.089193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:44.089204 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:24:44.089239 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:44.089240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:44.089250 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:44.089330 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 00:24:44.089492 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:44.090269 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:44.091855 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:44.092522 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:44.092619 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:44.092629 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:44.092937 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 00:24:44.092986 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 00:24:44.093580 19243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f8d19232d14e18b8181b4577725b0e" candidate_uuid: "e230f761c66040eba8399dcbd1ef5cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fcfb881df874ff194bde9da058d9dff" is_pre_election: true
I20260924 00:24:44.093552 19112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f8d19232d14e18b8181b4577725b0e" candidate_uuid: "e230f761c66040eba8399dcbd1ef5cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4790979c758e4af09a8f4e8bbc94a1f3" is_pre_election: true
I20260924 00:24:44.093755 19243 raft_consensus.cc:2454] T 01f8d19232d14e18b8181b4577725b0e P 2fcfb881df874ff194bde9da058d9dff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e230f761c66040eba8399dcbd1ef5cac in term 0.
I20260924 00:24:44.093761 19112 raft_consensus.cc:2454] T 01f8d19232d14e18b8181b4577725b0e P 4790979c758e4af09a8f4e8bbc94a1f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e230f761c66040eba8399dcbd1ef5cac in term 0.
I20260924 00:24:44.094044 19308 leader_election.cc:302] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fcfb881df874ff194bde9da058d9dff, e230f761c66040eba8399dcbd1ef5cac; no voters:
I20260924 00:24:44.094213 19440 raft_consensus.cc:2781] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 00:24:44.094287 19440 raft_consensus.cc:482] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 00:24:44.094324 19440 raft_consensus.cc:3037] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:44.095049 19440 raft_consensus.cc:504] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcfb881df874ff194bde9da058d9dff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33239 } } peers { permanent_uuid: "e230f761c66040eba8399dcbd1ef5cac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43793 } } peers { permanent_uuid: "4790979c758e4af09a8f4e8bbc94a1f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44989 } }
I20260924 00:24:44.095202 19440 leader_election.cc:288] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac [CANDIDATE]: Term 1 election: Requested vote from peers 2fcfb881df874ff194bde9da058d9dff (127.0.98.2:33239), 4790979c758e4af09a8f4e8bbc94a1f3 (127.0.98.1:44989)
I20260924 00:24:44.095410 19243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f8d19232d14e18b8181b4577725b0e" candidate_uuid: "e230f761c66040eba8399dcbd1ef5cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fcfb881df874ff194bde9da058d9dff"
I20260924 00:24:44.095506 19243 raft_consensus.cc:3037] T 01f8d19232d14e18b8181b4577725b0e P 2fcfb881df874ff194bde9da058d9dff [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:44.095500 19112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01f8d19232d14e18b8181b4577725b0e" candidate_uuid: "e230f761c66040eba8399dcbd1ef5cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4790979c758e4af09a8f4e8bbc94a1f3"
I20260924 00:24:44.095616 19112 raft_consensus.cc:3037] T 01f8d19232d14e18b8181b4577725b0e P 4790979c758e4af09a8f4e8bbc94a1f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:44.096352 19243 raft_consensus.cc:2454] T 01f8d19232d14e18b8181b4577725b0e P 2fcfb881df874ff194bde9da058d9dff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e230f761c66040eba8399dcbd1ef5cac in term 1.
I20260924 00:24:44.096520 19308 leader_election.cc:302] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fcfb881df874ff194bde9da058d9dff, e230f761c66040eba8399dcbd1ef5cac; no voters:
I20260924 00:24:44.096604 19112 raft_consensus.cc:2454] T 01f8d19232d14e18b8181b4577725b0e P 4790979c758e4af09a8f4e8bbc94a1f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e230f761c66040eba8399dcbd1ef5cac in term 1.
I20260924 00:24:44.096652 19440 raft_consensus.cc:2781] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:44.096845 19440 raft_consensus.cc:686] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac [term 1 LEADER]: Becoming Leader. State: Replica: e230f761c66040eba8399dcbd1ef5cac, State: Running, Role: LEADER
I20260924 00:24:44.096998 19440 consensus_queue.cc:237] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcfb881df874ff194bde9da058d9dff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33239 } } peers { permanent_uuid: "e230f761c66040eba8399dcbd1ef5cac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43793 } } peers { permanent_uuid: "4790979c758e4af09a8f4e8bbc94a1f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44989 } }
I20260924 00:24:44.098030 18968 catalog_manager.cc:5495] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac reported cstate change: term changed from 0 to 1, leader changed from <none> to e230f761c66040eba8399dcbd1ef5cac (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e230f761c66040eba8399dcbd1ef5cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fcfb881df874ff194bde9da058d9dff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e230f761c66040eba8399dcbd1ef5cac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4790979c758e4af09a8f4e8bbc94a1f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44989 } health_report { overall_health: UNKNOWN } } }
I20260924 00:24:44.110942 19430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:44.108488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36590 (local address 127.0.98.3:43793)
0924 00:24:44.108554 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:44.108556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:44.108566 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:24:44.108614 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:44.108615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:44.108632 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:44.108708 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:24:44.108838 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:44.109339 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:44.110125 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:44.110333 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:44.110402 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:44.110581 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:24:44.110689 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:24:44.110786 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:44.110803 (+ 17us) server_negotiation.cc:299] Negotiation successful
0924 00:24:44.110821 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 00:24:44.113134 19112 raft_consensus.cc:1261] T 01f8d19232d14e18b8181b4577725b0e P 4790979c758e4af09a8f4e8bbc94a1f3 [term 1 FOLLOWER]: Refusing update from remote peer e230f761c66040eba8399dcbd1ef5cac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:24:44.113134 19243 raft_consensus.cc:1261] T 01f8d19232d14e18b8181b4577725b0e P 2fcfb881df874ff194bde9da058d9dff [term 1 FOLLOWER]: Refusing update from remote peer e230f761c66040eba8399dcbd1ef5cac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:24:44.113554 19444 consensus_queue.cc:1035] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4790979c758e4af09a8f4e8bbc94a1f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:24:44.113754 19440 consensus_queue.cc:1035] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fcfb881df874ff194bde9da058d9dff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:24:44.116566 19447 mvcc.cc:205] Tried to move back new op lower bound from 7332698046923939840 to 7332698046862286848. Current Snapshot: MvccSnapshot[applied={T|T < 7332698046923939840}]
I20260924 00:24:44.119064 18968 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:24:44.119169 18968 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 00:24:44.120059 18968 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d1187a11d7741cc8539987b38812dae: Sending DeleteTablet for 3 replicas of tablet 01f8d19232d14e18b8181b4577725b0e
I20260924 00:24:44.120515 19354 tablet_service.cc:1514] Processing DeleteTablet for tablet 01f8d19232d14e18b8181b4577725b0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:24:44 UTC) from {username='slave'} at 127.0.0.1:36574
I20260924 00:24:44.120522 19223 tablet_service.cc:1514] Processing DeleteTablet for tablet 01f8d19232d14e18b8181b4577725b0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:24:44 UTC) from {username='slave'} at 127.0.0.1:43830
I20260924 00:24:44.120558 19092 tablet_service.cc:1514] Processing DeleteTablet for tablet 01f8d19232d14e18b8181b4577725b0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 00:24:44 UTC) from {username='slave'} at 127.0.0.1:34934
I20260924 00:24:44.120807 19457 tablet_replica.cc:330] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac: stopping tablet replica
I20260924 00:24:44.120944 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 19030
I20260924 00:24:44.124248 19457 raft_consensus.cc:2229] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac [term 1 LEADER]: Raft consensus shutting down.
I20260924 00:24:44.124473 19457 raft_consensus.cc:2258] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:24:44.124706 19456 tablet_replica.cc:330] T 01f8d19232d14e18b8181b4577725b0e P 2fcfb881df874ff194bde9da058d9dff: stopping tablet replica
I20260924 00:24:44.124889 19456 raft_consensus.cc:2229] T 01f8d19232d14e18b8181b4577725b0e P 2fcfb881df874ff194bde9da058d9dff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 00:24:44.125069 19456 raft_consensus.cc:2258] T 01f8d19232d14e18b8181b4577725b0e P 2fcfb881df874ff194bde9da058d9dff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 00:24:44.125174 19457 ts_tablet_manager.cc:1905] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 00:24:44.125677 19456 ts_tablet_manager.cc:1905] T 01f8d19232d14e18b8181b4577725b0e P 2fcfb881df874ff194bde9da058d9dff: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 00:24:44.127358 18955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44989 (error 108)
I20260924 00:24:44.127429 19457 ts_tablet_manager.cc:1918] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:24:44.127504 19457 log.cc:1196] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-2/wal/wals/01f8d19232d14e18b8181b4577725b0e
I20260924 00:24:44.127873 19457 ts_tablet_manager.cc:1939] T 01f8d19232d14e18b8181b4577725b0e P e230f761c66040eba8399dcbd1ef5cac: Deleting consensus metadata
I20260924 00:24:44.127911 19456 ts_tablet_manager.cc:1918] T 01f8d19232d14e18b8181b4577725b0e P 2fcfb881df874ff194bde9da058d9dff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 00:24:44.127978 19456 log.cc:1196] T 01f8d19232d14e18b8181b4577725b0e P 2fcfb881df874ff194bde9da058d9dff: Deleting WAL directory at /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790209154318147-392-0/minicluster-data/ts-1/wal/wals/01f8d19232d14e18b8181b4577725b0e
I20260924 00:24:44.128201 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 19161
I20260924 00:24:44.128903 19432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:44.128460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44989 (local address 127.0.0.1:34936)
0924 00:24:44.128808 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:44.128855 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327}
W20260924 00:24:44.129030 18955 catalog_manager.cc:4584] TS 4790979c758e4af09a8f4e8bbc94a1f3 (127.0.98.1:44989): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01f8d19232d14e18b8181b4577725b0e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44989: connect: Connection refused (error 111)
I20260924 00:24:44.129420 18954 catalog_manager.cc:4841] TS e230f761c66040eba8399dcbd1ef5cac (127.0.98.3:43793): tablet 01f8d19232d14e18b8181b4577725b0e (table test-table [id=97b75aaa8aae470980ebd340b0d3d6cb]) successfully deleted
I20260924 00:24:44.135260 19432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:44.135093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33239 (local address 127.0.0.1:43838)
0924 00:24:44.135169 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0924 00:24:44.135206 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 00:24:44.135264 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 19292
I20260924 00:24:44.140098 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 18939
2026-09-24T00:24:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (843 ms)
[----------] 6 tests from EncryptionPolicyTest (8783 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:24:44 dist-test-slave-777x krb5kdc[19462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:24:44 dist-test-slave-777x krb5kdc[19462](info): set up 2 sockets
Sep 24 00:24:44 dist-test-slave-777x krb5kdc[19462](info): commencing operation
krb5kdc: starting...
W20260924 00:24:45.208654 392 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:24:45 dist-test-slave-777x krb5kdc[19462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:24:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:24:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:24:45.343442 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36377
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:45.433684 19475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:45.433945 19475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:45.433997 19475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:45.435176 19475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:24:45.435245 19475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:45.435266 19475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:45.435287 19475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:24:45.435316 19475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:24:45.436829 19475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36377
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:45.437120 19475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:45.437348 19475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:45.442574 19483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:45.442581 19480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:45.442607 19475 server_base.cc:1034] running on GCE node
W20260924 00:24:45.442574 19481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:45.443053 19475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:45.443387 19475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:45.444548 19475 hybrid_clock.cc:648] HybridClock initialized: now 1790209485444548 us; error 53 us; skew 500 ppm
Sep 24 00:24:45 dist-test-slave-777x krb5kdc[19462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:45.447218 19475 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:24:45.447669 19475 webserver.cc:460] Webserver started at http://127.0.0.1:34089/ using document root <none> and password file <none>
I20260924 00:24:45.447885 19475 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:45.447949 19475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:45.448091 19475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:45.449218 19475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "719ce08cf4ab4cf69d9e299025178dd4"
format_stamp: "Formatted at 2026-09-24 00:24:45 on dist-test-slave-777x"
server_key: "fc909fc92722088bc2a292fa149b8362"
server_key_iv: "68dc96d5e03dd1200d33893108a9ad9a"
server_key_version: "encryptionkey@0"
I20260924 00:24:45.449594 19475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "719ce08cf4ab4cf69d9e299025178dd4"
format_stamp: "Formatted at 2026-09-24 00:24:45 on dist-test-slave-777x"
server_key: "fc909fc92722088bc2a292fa149b8362"
server_key_iv: "68dc96d5e03dd1200d33893108a9ad9a"
server_key_version: "encryptionkey@0"
I20260924 00:24:45.451054 19475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:24:45.452014 19490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:45.452265 19475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:45.452369 19475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "719ce08cf4ab4cf69d9e299025178dd4"
format_stamp: "Formatted at 2026-09-24 00:24:45 on dist-test-slave-777x"
server_key: "fc909fc92722088bc2a292fa149b8362"
server_key_iv: "68dc96d5e03dd1200d33893108a9ad9a"
server_key_version: "encryptionkey@0"
I20260924 00:24:45.452456 19475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:45.476104 19475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:45.478587 19475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:45.478765 19475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:45.482722 19475 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36377
I20260924 00:24:45.482766 19542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36377 every 8 connection(s)
I20260924 00:24:45.483166 19475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:24:45.484130 19543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:45.486611 19543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4: Bootstrap starting.
I20260924 00:24:45.487315 19543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:45.487615 19543 log.cc:824] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:45.488474 19543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4: No bootstrap required, opened a new log
I20260924 00:24:45.488884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 19475
I20260924 00:24:45.489018 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:24:45.489212 392 external_mini_cluster.cc:1428] Setting key d6bab5e30d0822a1e888b8d03eb1a948
I20260924 00:24:45.490986 19543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719ce08cf4ab4cf69d9e299025178dd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36377 } }
I20260924 00:24:45.491187 19543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:45.491230 19543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 719ce08cf4ab4cf69d9e299025178dd4, State: Initialized, Role: FOLLOWER
I20260924 00:24:45.491427 19543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719ce08cf4ab4cf69d9e299025178dd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36377 } }
I20260924 00:24:45.491523 19543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:24:45.491550 19543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:24:45.491626 19543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:45.492960 19543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719ce08cf4ab4cf69d9e299025178dd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36377 } }
I20260924 00:24:45.493104 19543 leader_election.cc:302] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 719ce08cf4ab4cf69d9e299025178dd4; no voters:
I20260924 00:24:45.493292 19543 leader_election.cc:288] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:24:45.493359 19548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:45.493570 19543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:24:45.493574 19548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4 [term 1 LEADER]: Becoming Leader. State: Replica: 719ce08cf4ab4cf69d9e299025178dd4, State: Running, Role: LEADER
I20260924 00:24:45.493919 19548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719ce08cf4ab4cf69d9e299025178dd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36377 } }
I20260924 00:24:45.494349 19549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "719ce08cf4ab4cf69d9e299025178dd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719ce08cf4ab4cf69d9e299025178dd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36377 } } }
I20260924 00:24:45.494457 19549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4 [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:45.494697 19550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 719ce08cf4ab4cf69d9e299025178dd4. Latest consensus state: current_term: 1 leader_uuid: "719ce08cf4ab4cf69d9e299025178dd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "719ce08cf4ab4cf69d9e299025178dd4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36377 } } }
I20260924 00:24:45.494841 19550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4 [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:45.495050 19557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:24:45.495673 19557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:24:45.497573 19557 catalog_manager.cc:1347] Generated new cluster ID: 6e93b9dd0b9844c5b1e2e74ede192128
I20260924 00:24:45.497637 19557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:24:45 dist-test-slave-777x krb5kdc[19462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 00:24:45.502912 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:45.489830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47797 (local address 127.0.0.1:36377)
0924 00:24:45.490071 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:45.490082 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:45.490118 (+ 36us) server_negotiation.cc:407] Connection header received
0924 00:24:45.490941 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:45.490950 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:45.491022 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:45.491190 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:45.492012 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:45.493557 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:45.494878 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:45.495526 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:45.495633 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:45.498797 (+ 3164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:45.498802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:45.498802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:45.498814 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:45.500984 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:45.501488 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:45.501492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:45.501492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:45.501633 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:45.501875 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:45.501876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:45.501876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:45.502262 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:45.502467 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:45.502561 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 00:24:45.502673 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":116,"threads_started":1}
I20260924 00:24:45.504060 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:24:45.513571 19547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:45.505624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42463 (local address 127.0.0.1:36377)
0924 00:24:45.505663 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:45.505665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:45.505800 (+ 135us) server_negotiation.cc:407] Connection header received
0924 00:24:45.505898 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:45.505898 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:45.505908 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:45.505937 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:45.506665 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:45.507417 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:45.508521 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:45.508851 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:45.508893 (+ 42us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:45.510551 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:45.510553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:45.510554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:45.510561 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:45.512228 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:45.512670 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:45.512672 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:45.512672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:45.512725 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:45.512971 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:45.512972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:45.512972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:45.513101 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:45.513174 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:45.513403 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 00:24:45.513468 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 00:24:45.515436 19557 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:24:45.516208 19557 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:24:45.518968 19557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 719ce08cf4ab4cf69d9e299025178dd4: Generated new TSK 0
I20260924 00:24:45.519150 19557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:24:45.524770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 19475
W20260924 00:24:45.529232 19460 connection.cc:535] client connection to 127.0.0.1:36377 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36377): Connection reset by peer (error 104)
2026-09-24T00:24:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1382 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:24:45 dist-test-slave-777x krb5kdc[19574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:24:45 dist-test-slave-777x krb5kdc[19574](info): set up 2 sockets
Sep 24 00:24:45 dist-test-slave-777x krb5kdc[19574](info): commencing operation
krb5kdc: starting...
W20260924 00:24:46.640347 392 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:24:46 dist-test-slave-777x krb5kdc[19574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:24:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:24:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:24:46.764045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38563
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:46.862013 19587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:46.862308 19587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:46.862388 19587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:46.863752 19587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:24:46.863811 19587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:46.863838 19587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:46.863870 19587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:24:46.863889 19587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:24:46.866218 19587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38563
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:46.866513 19587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:46.866766 19587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:46.871781 19595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:46.871781 19592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:46.872097 19593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:46.872433 19587 server_base.cc:1034] running on GCE node
I20260924 00:24:46.872704 19587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:46.873039 19587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:46.874236 19587 hybrid_clock.cc:648] HybridClock initialized: now 1790209486874205 us; error 44 us; skew 500 ppm
Sep 24 00:24:46 dist-test-slave-777x krb5kdc[19574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209486, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:46.877295 19587 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:24:46.877787 19587 webserver.cc:460] Webserver started at http://127.0.0.1:42703/ using document root <none> and password file <none>
I20260924 00:24:46.878058 19587 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:46.878125 19587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:46.878263 19587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:46.879367 19587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "b77963ab820d40c085a6dd2419389fd7"
format_stamp: "Formatted at 2026-09-24 00:24:46 on dist-test-slave-777x"
server_key: "659aa1f33de4050c8aa74657f09f30b8"
server_key_iv: "68f90fca01fcc3aabc47ebffe6bb4f39"
server_key_version: "encryptionkey@0"
I20260924 00:24:46.879751 19587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "b77963ab820d40c085a6dd2419389fd7"
format_stamp: "Formatted at 2026-09-24 00:24:46 on dist-test-slave-777x"
server_key: "659aa1f33de4050c8aa74657f09f30b8"
server_key_iv: "68f90fca01fcc3aabc47ebffe6bb4f39"
server_key_version: "encryptionkey@0"
I20260924 00:24:46.881249 19587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:24:46.882079 19602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:46.882277 19587 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:24:46.882375 19587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "b77963ab820d40c085a6dd2419389fd7"
format_stamp: "Formatted at 2026-09-24 00:24:46 on dist-test-slave-777x"
server_key: "659aa1f33de4050c8aa74657f09f30b8"
server_key_iv: "68f90fca01fcc3aabc47ebffe6bb4f39"
server_key_version: "encryptionkey@0"
I20260924 00:24:46.882464 19587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:46.907140 19587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:46.909499 19587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:46.909679 19587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:46.913784 19587 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38563
I20260924 00:24:46.913852 19654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38563 every 8 connection(s)
I20260924 00:24:46.914232 19587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:24:46.915195 19655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:46.917652 19655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7: Bootstrap starting.
I20260924 00:24:46.918258 19655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:46.918541 19655 log.cc:824] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:46.919330 19655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7: No bootstrap required, opened a new log
I20260924 00:24:46.919943 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 19587
I20260924 00:24:46.920083 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:24:46.920322 392 external_mini_cluster.cc:1428] Setting key 4fb08bd917ce2f26a08d6c7ddab51a92
I20260924 00:24:46.921837 19655 raft_consensus.cc:348] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77963ab820d40c085a6dd2419389fd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38563 } }
I20260924 00:24:46.922050 19655 raft_consensus.cc:374] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:46.922086 19655 raft_consensus.cc:729] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b77963ab820d40c085a6dd2419389fd7, State: Initialized, Role: FOLLOWER
I20260924 00:24:46.922211 19655 consensus_queue.cc:260] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77963ab820d40c085a6dd2419389fd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38563 } }
I20260924 00:24:46.922302 19655 raft_consensus.cc:388] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:24:46.922339 19655 raft_consensus.cc:482] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:24:46.922427 19655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:46.923017 19655 raft_consensus.cc:504] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77963ab820d40c085a6dd2419389fd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38563 } }
I20260924 00:24:46.923141 19655 leader_election.cc:302] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b77963ab820d40c085a6dd2419389fd7; no voters:
I20260924 00:24:46.923341 19655 leader_election.cc:288] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:24:46.923370 19660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:46.923480 19660 raft_consensus.cc:686] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7 [term 1 LEADER]: Becoming Leader. State: Replica: b77963ab820d40c085a6dd2419389fd7, State: Running, Role: LEADER
I20260924 00:24:46.923635 19655 sys_catalog.cc:564] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:24:46.923637 19660 consensus_queue.cc:237] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77963ab820d40c085a6dd2419389fd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38563 } }
I20260924 00:24:46.924124 19662 sys_catalog.cc:455] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b77963ab820d40c085a6dd2419389fd7. Latest consensus state: current_term: 1 leader_uuid: "b77963ab820d40c085a6dd2419389fd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77963ab820d40c085a6dd2419389fd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38563 } } }
I20260924 00:24:46.924336 19662 sys_catalog.cc:458] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7 [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:46.924130 19661 sys_catalog.cc:455] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b77963ab820d40c085a6dd2419389fd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b77963ab820d40c085a6dd2419389fd7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38563 } } }
I20260924 00:24:46.924588 19661 sys_catalog.cc:458] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7 [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:46.924604 19666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:24:46.925292 19666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:24:46.927325 19666 catalog_manager.cc:1347] Generated new cluster ID: 33fd793bc6ea4cb692cf1d37634b390a
I20260924 00:24:46.927392 19666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:24:46.928303 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:46.920935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55131 (local address 127.0.0.1:38563)
0924 00:24:46.921128 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:46.921139 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:46.921173 (+ 34us) server_negotiation.cc:407] Connection header received
0924 00:24:46.922035 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:46.922067 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:46.922135 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:46.922296 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:46.922624 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:46.924110 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:46.925480 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:46.926266 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:46.926442 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:46.927331 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:46.927336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:46.927336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:46.927593 (+ 257us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:46.927637 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:46.927653 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:46.927935 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:46.927984 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 00:24:46.928015 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
I20260924 00:24:46.933461 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:24:46.940121 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:46.935160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53075 (local address 127.0.0.1:38563)
0924 00:24:46.935207 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:46.935209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:46.936385 (+ 1176us) server_negotiation.cc:407] Connection header received
0924 00:24:46.936463 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:46.936464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:46.936477 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:46.936546 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:46.936703 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:46.937485 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:46.938605 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:46.938940 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:46.939013 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:46.939820 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:46.939822 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:46.939822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:46.939827 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:46.939853 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:46.939858 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:46.939921 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:46.940020 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 00:24:46.940032 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:24:46.948050 19666 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:24:46.948801 19666 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:24:46.954221 19666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b77963ab820d40c085a6dd2419389fd7: Generated new TSK 0
I20260924 00:24:46.954493 19666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:24:46.958040 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 19587
W20260924 00:24:46.961990 19572 connection.cc:535] client connection to 127.0.0.1:38563 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38563): Connection reset by peer (error 104)
2026-09-24T00:24:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:24:47 dist-test-slave-777x krb5kdc[19686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:24:47 dist-test-slave-777x krb5kdc[19686](info): set up 2 sockets
Sep 24 00:24:47 dist-test-slave-777x krb5kdc[19686](info): commencing operation
krb5kdc: starting...
W20260924 00:24:48.028868 392 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:24:48 dist-test-slave-777x krb5kdc[19686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209488, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:24:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:24:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:24:48.155746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38953
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:48.240214 19699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:48.240444 19699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:48.240480 19699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:48.241694 19699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:24:48.241748 19699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:48.241762 19699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:48.241777 19699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:24:48.241789 19699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:24:48.243301 19699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38953
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:48.243566 19699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:48.243800 19699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:48.248781 19704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:48.248798 19707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:48.248832 19705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:48.249140 19699 server_base.cc:1034] running on GCE node
I20260924 00:24:48.249411 19699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:48.249708 19699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:48.250859 19699 hybrid_clock.cc:648] HybridClock initialized: now 1790209488250824 us; error 48 us; skew 500 ppm
Sep 24 00:24:48 dist-test-slave-777x krb5kdc[19686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209488, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:48.253700 19699 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:24:48.254231 19699 webserver.cc:460] Webserver started at http://127.0.0.1:34581/ using document root <none> and password file <none>
I20260924 00:24:48.254485 19699 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:48.254560 19699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:48.254721 19699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:48.255957 19699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "359344281b2d47b1b2bb9843ef5831b7"
format_stamp: "Formatted at 2026-09-24 00:24:48 on dist-test-slave-777x"
server_key: "be25e3a66a1ab9141d5e128f4a002217"
server_key_iv: "0323bb42f6106860bf863b8635d89bbd"
server_key_version: "encryptionkey@0"
I20260924 00:24:48.256397 19699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "359344281b2d47b1b2bb9843ef5831b7"
format_stamp: "Formatted at 2026-09-24 00:24:48 on dist-test-slave-777x"
server_key: "be25e3a66a1ab9141d5e128f4a002217"
server_key_iv: "0323bb42f6106860bf863b8635d89bbd"
server_key_version: "encryptionkey@0"
I20260924 00:24:48.257973 19699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:24:48.258821 19714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:48.259141 19699 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:48.259265 19699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "359344281b2d47b1b2bb9843ef5831b7"
format_stamp: "Formatted at 2026-09-24 00:24:48 on dist-test-slave-777x"
server_key: "be25e3a66a1ab9141d5e128f4a002217"
server_key_iv: "0323bb42f6106860bf863b8635d89bbd"
server_key_version: "encryptionkey@0"
I20260924 00:24:48.259369 19699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:48.262914 19699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:48.265213 19699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:48.265406 19699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:48.269662 19699 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38953
I20260924 00:24:48.269712 19766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38953 every 8 connection(s)
I20260924 00:24:48.270079 19699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:24:48.271061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 19699
I20260924 00:24:48.271374 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:24:48.271626 392 external_mini_cluster.cc:1428] Setting key 940fc98c4030933e377438a5602a083d
I20260924 00:24:48.271756 19767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:48.274750 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:48.272279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57427 (local address 127.0.0.1:38953)
0924 00:24:48.272467 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:48.272476 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:48.272510 (+ 34us) server_negotiation.cc:407] Connection header received
0924 00:24:48.273263 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:48.273271 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:48.273300 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:48.273496 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:48.274284 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:48.274287 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:48.274288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:48.274292 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:48.274331 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:48.274342 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:48.274399 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:48.274488 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 00:24:48.274554 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260924 00:24:48.275871 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:24:48.275870 19767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7: Bootstrap starting.
I20260924 00:24:48.276674 19767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:48.277087 19767 log.cc:824] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:48.278116 19767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7: No bootstrap required, opened a new log
I20260924 00:24:48.278800 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:48.277623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56831 (local address 127.0.0.1:38953)
0924 00:24:48.277664 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:48.277666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:48.277792 (+ 126us) server_negotiation.cc:407] Connection header received
0924 00:24:48.277853 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:48.277854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:48.277859 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:48.277906 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:48.278544 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:48.278545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:48.278545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:48.278549 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:48.278563 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:48.278567 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:48.278620 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:48.278684 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 00:24:48.278696 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:24:48.280381 19767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359344281b2d47b1b2bb9843ef5831b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38953 } }
I20260924 00:24:48.280544 19767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:48.280594 19767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359344281b2d47b1b2bb9843ef5831b7, State: Initialized, Role: FOLLOWER
I20260924 00:24:48.280731 19767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359344281b2d47b1b2bb9843ef5831b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38953 } }
I20260924 00:24:48.280823 19767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:24:48.280881 19767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:24:48.280925 19767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:48.281582 19767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359344281b2d47b1b2bb9843ef5831b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38953 } }
I20260924 00:24:48.281694 19767 leader_election.cc:302] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 359344281b2d47b1b2bb9843ef5831b7; no voters:
I20260924 00:24:48.281894 19767 leader_election.cc:288] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:24:48.281970 19777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:48.282127 19777 raft_consensus.cc:686] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7 [term 1 LEADER]: Becoming Leader. State: Replica: 359344281b2d47b1b2bb9843ef5831b7, State: Running, Role: LEADER
I20260924 00:24:48.282181 19767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:24:48.282233 19777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359344281b2d47b1b2bb9843ef5831b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38953 } }
I20260924 00:24:48.282656 19778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "359344281b2d47b1b2bb9843ef5831b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359344281b2d47b1b2bb9843ef5831b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38953 } } }
I20260924 00:24:48.282838 19778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7 [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:48.282903 19779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 359344281b2d47b1b2bb9843ef5831b7. Latest consensus state: current_term: 1 leader_uuid: "359344281b2d47b1b2bb9843ef5831b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "359344281b2d47b1b2bb9843ef5831b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38953 } } }
I20260924 00:24:48.283044 19779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7 [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:48.283109 19784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:24:48.283615 19784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:24:48.285254 19784 catalog_manager.cc:1347] Generated new cluster ID: f11c1c4889c24b5e94c8938629d67c89
I20260924 00:24:48.285367 19784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:24:48.305238 19784 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:24:48.305508 19784 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:24:48.308630 19784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 359344281b2d47b1b2bb9843ef5831b7: Generated new TSK 0
I20260924 00:24:48.308840 19784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:24:48.318132 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 19699
2026-09-24T00:24:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:24:48 dist-test-slave-777x krb5kdc[19798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:24:48 dist-test-slave-777x krb5kdc[19798](info): set up 2 sockets
Sep 24 00:24:48 dist-test-slave-777x krb5kdc[19798](info): commencing operation
krb5kdc: starting...
W20260924 00:24:49.452247 392 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:24:49 dist-test-slave-777x krb5kdc[19798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:24:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:24:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:24:49.577294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37229
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:49.661307 19811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:49.661574 19811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:49.661616 19811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:49.662842 19811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:24:49.662925 19811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:49.662949 19811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:49.662971 19811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:24:49.662998 19811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:24:49.664608 19811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:49.664917 19811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:49.665162 19811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:49.670279 19819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:49.670354 19817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:49.670279 19816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:49.670625 19811 server_base.cc:1034] running on GCE node
I20260924 00:24:49.670877 19811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:49.671221 19811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:49.672379 19811 hybrid_clock.cc:648] HybridClock initialized: now 1790209489672357 us; error 34 us; skew 500 ppm
Sep 24 00:24:49 dist-test-slave-777x krb5kdc[19798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209489, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:49.675061 19811 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:24:49.675458 19811 webserver.cc:460] Webserver started at http://127.0.0.1:41531/ using document root <none> and password file <none>
I20260924 00:24:49.675661 19811 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:49.675704 19811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:49.675791 19811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:49.676913 19811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "1db8bbf621c745bbac56b6fd8b831d5c"
format_stamp: "Formatted at 2026-09-24 00:24:49 on dist-test-slave-777x"
server_key: "ea092aab0cac7977508b1992185c50c7"
server_key_iv: "d21a4493ba7144359dd22b32858c4b9a"
server_key_version: "encryptionkey@0"
I20260924 00:24:49.677234 19811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "1db8bbf621c745bbac56b6fd8b831d5c"
format_stamp: "Formatted at 2026-09-24 00:24:49 on dist-test-slave-777x"
server_key: "ea092aab0cac7977508b1992185c50c7"
server_key_iv: "d21a4493ba7144359dd22b32858c4b9a"
server_key_version: "encryptionkey@0"
I20260924 00:24:49.678542 19811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:24:49.679430 19826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:49.679641 19811 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:49.679741 19811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "1db8bbf621c745bbac56b6fd8b831d5c"
format_stamp: "Formatted at 2026-09-24 00:24:49 on dist-test-slave-777x"
server_key: "ea092aab0cac7977508b1992185c50c7"
server_key_iv: "d21a4493ba7144359dd22b32858c4b9a"
server_key_version: "encryptionkey@0"
I20260924 00:24:49.679816 19811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:49.698740 19811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:49.701488 19811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:49.701661 19811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:49.706139 19811 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37229
I20260924 00:24:49.706202 19878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37229 every 8 connection(s)
I20260924 00:24:49.706581 19811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:24:49.707643 19879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:49.710003 19879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c: Bootstrap starting.
I20260924 00:24:49.710544 19879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:49.710776 19879 log.cc:824] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:49.711467 19879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c: No bootstrap required, opened a new log
I20260924 00:24:49.712983 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 19811
I20260924 00:24:49.713116 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:24:49.713117 19879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db8bbf621c745bbac56b6fd8b831d5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37229 } }
I20260924 00:24:49.713295 19879 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:49.713318 392 external_mini_cluster.cc:1428] Setting key c02300812686535d7aa133b832767aed
I20260924 00:24:49.713366 19879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1db8bbf621c745bbac56b6fd8b831d5c, State: Initialized, Role: FOLLOWER
I20260924 00:24:49.713521 19879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db8bbf621c745bbac56b6fd8b831d5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37229 } }
I20260924 00:24:49.713613 19879 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:24:49.713649 19879 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:24:49.713711 19879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:49.714331 19879 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db8bbf621c745bbac56b6fd8b831d5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37229 } }
I20260924 00:24:49.714458 19879 leader_election.cc:302] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1db8bbf621c745bbac56b6fd8b831d5c; no voters:
I20260924 00:24:49.714651 19879 leader_election.cc:288] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:24:49.714679 19884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:49.714789 19884 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c [term 1 LEADER]: Becoming Leader. State: Replica: 1db8bbf621c745bbac56b6fd8b831d5c, State: Running, Role: LEADER
I20260924 00:24:49.714911 19879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:24:49.714942 19884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db8bbf621c745bbac56b6fd8b831d5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37229 } }
I20260924 00:24:49.715345 19884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1db8bbf621c745bbac56b6fd8b831d5c. Latest consensus state: current_term: 1 leader_uuid: "1db8bbf621c745bbac56b6fd8b831d5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db8bbf621c745bbac56b6fd8b831d5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37229 } } }
I20260924 00:24:49.715610 19884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:49.715538 19886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1db8bbf621c745bbac56b6fd8b831d5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db8bbf621c745bbac56b6fd8b831d5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37229 } } }
I20260924 00:24:49.716069 19886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:49.716246 19893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:24:49.716753 19893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:24:49.718562 19893 catalog_manager.cc:1347] Generated new cluster ID: 26f15c89b9d94f46b7017e16c712ab85
I20260924 00:24:49.718621 19893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:24:49 dist-test-slave-777x krb5kdc[19798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209489, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 00:24:49.725564 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:49.714073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44863 (local address 127.0.0.1:37229)
0924 00:24:49.714592 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:49.714603 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:49.714639 (+ 36us) server_negotiation.cc:407] Connection header received
0924 00:24:49.714768 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:49.714775 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:49.714841 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:49.715863 (+ 1022us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:49.715996 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:49.717555 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:49.718711 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:49.719185 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:49.719312 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:49.721908 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:49.721914 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:49.721914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:49.721933 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:49.723796 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:49.724217 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:49.724221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:49.724221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:49.724330 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:49.724557 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:49.724559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:49.724559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:49.724842 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:49.725037 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:49.725121 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 00:24:49.725384 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":431,"threads_started":1}
I20260924 00:24:49.726608 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:24:49.726858 19893 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:24:49.727658 19893 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:24:49.736954 19893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1db8bbf621c745bbac56b6fd8b831d5c: Generated new TSK 0
I20260924 00:24:49.737210 19893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:24:49.739879 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:49.732612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51757 (local address 127.0.0.1:37229)
0924 00:24:49.732660 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:49.732662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:49.732803 (+ 141us) server_negotiation.cc:407] Connection header received
0924 00:24:49.732910 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:49.732911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:49.732924 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:49.733010 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:49.733617 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:49.734232 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:49.735079 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:49.735319 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:49.735408 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:49.737040 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:49.737041 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:49.737041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:49.737050 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:49.738696 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:49.739071 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:49.739073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:49.739073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:49.739134 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:49.739336 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:49.739337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:49.739338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:49.739460 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:49.739570 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:49.739678 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 00:24:49.739769 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:24:49.742189 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 19811
2026-09-24T00:24:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:24:49 dist-test-slave-777x krb5kdc[19910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:24:49 dist-test-slave-777x krb5kdc[19910](info): set up 2 sockets
Sep 24 00:24:49 dist-test-slave-777x krb5kdc[19910](info): commencing operation
krb5kdc: starting...
W20260924 00:24:50.792328 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:24:50 dist-test-slave-777x krb5kdc[19910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:24:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:24:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:24:50.922776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46523
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:51.013314 19923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:51.013548 19923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:51.013574 19923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:51.014920 19923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:24:51.014983 19923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:51.015000 19923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:51.015014 19923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:24:51.015028 19923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:24:51.016883 19923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:51.017189 19923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:51.017478 19923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:51.022985 19929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:51.022997 19931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:51.023000 19928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:51.023471 19923 server_base.cc:1034] running on GCE node
I20260924 00:24:51.023778 19923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:51.024123 19923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:51.025350 19923 hybrid_clock.cc:648] HybridClock initialized: now 1790209491025331 us; error 31 us; skew 500 ppm
Sep 24 00:24:51 dist-test-slave-777x krb5kdc[19910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:51.028081 19923 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:24:51.028671 19923 webserver.cc:460] Webserver started at http://127.0.0.1:40861/ using document root <none> and password file <none>
I20260924 00:24:51.028935 19923 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:51.029006 19923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:51.029158 19923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:51.030346 19923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "0e4c51f96c38492db7f269b964f7cf5a"
format_stamp: "Formatted at 2026-09-24 00:24:51 on dist-test-slave-777x"
server_key: "7a3fb5126f4c77479dfcff2e59db5e28"
server_key_iv: "7e698af17306ffeae438405b6b1d8594"
server_key_version: "encryptionkey@0"
I20260924 00:24:51.030757 19923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "0e4c51f96c38492db7f269b964f7cf5a"
format_stamp: "Formatted at 2026-09-24 00:24:51 on dist-test-slave-777x"
server_key: "7a3fb5126f4c77479dfcff2e59db5e28"
server_key_iv: "7e698af17306ffeae438405b6b1d8594"
server_key_version: "encryptionkey@0"
I20260924 00:24:51.032378 19923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260924 00:24:51.033326 19938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:51.033524 19923 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:51.033640 19923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "0e4c51f96c38492db7f269b964f7cf5a"
format_stamp: "Formatted at 2026-09-24 00:24:51 on dist-test-slave-777x"
server_key: "7a3fb5126f4c77479dfcff2e59db5e28"
server_key_iv: "7e698af17306ffeae438405b6b1d8594"
server_key_version: "encryptionkey@0"
I20260924 00:24:51.033728 19923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:51.058820 19923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:51.061355 19923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:51.061540 19923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:51.066351 19923 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46523
I20260924 00:24:51.066437 19990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46523 every 8 connection(s)
I20260924 00:24:51.066845 19923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:24:51.067822 19991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:51.068346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 19923
I20260924 00:24:51.068482 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:24:51.068665 392 external_mini_cluster.cc:1428] Setting key 50159f3845665d6db7d6d50473f17402
I20260924 00:24:51.070667 19991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a: Bootstrap starting.
I20260924 00:24:51.071251 19991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:51.071552 19991 log.cc:824] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:51.072249 19991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a: No bootstrap required, opened a new log
I20260924 00:24:51.074198 19991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e4c51f96c38492db7f269b964f7cf5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } }
I20260924 00:24:51.074357 19991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:51.074404 19991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e4c51f96c38492db7f269b964f7cf5a, State: Initialized, Role: FOLLOWER
I20260924 00:24:51.074576 19991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e4c51f96c38492db7f269b964f7cf5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } }
I20260924 00:24:51.074667 19991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:24:51.074728 19991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:24:51.074779 19991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:51.075098 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:51.069317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33489 (local address 127.0.0.1:46523)
0924 00:24:51.069521 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:51.069531 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:51.069567 (+ 36us) server_negotiation.cc:407] Connection header received
0924 00:24:51.070036 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:51.070043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:51.070095 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:51.070250 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:51.070551 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:51.071685 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:51.072731 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:51.073215 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:51.073359 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:51.074195 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:51.074206 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:51.074206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:51.074433 (+ 227us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:51.074504 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:51.074523 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:51.074610 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:51.074708 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 00:24:51.074916 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
I20260924 00:24:51.075395 19991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e4c51f96c38492db7f269b964f7cf5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } }
I20260924 00:24:51.075512 19991 leader_election.cc:302] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e4c51f96c38492db7f269b964f7cf5a; no voters:
I20260924 00:24:51.075670 19991 leader_election.cc:288] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:24:51.075698 19996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:51.075856 19996 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a [term 1 LEADER]: Becoming Leader. State: Replica: 0e4c51f96c38492db7f269b964f7cf5a, State: Running, Role: LEADER
I20260924 00:24:51.075934 19991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:24:51.075954 19996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e4c51f96c38492db7f269b964f7cf5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } }
I20260924 00:24:51.076190 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:24:51.076330 19996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e4c51f96c38492db7f269b964f7cf5a. Latest consensus state: current_term: 1 leader_uuid: "0e4c51f96c38492db7f269b964f7cf5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e4c51f96c38492db7f269b964f7cf5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } } }
I20260924 00:24:51.076344 19998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e4c51f96c38492db7f269b964f7cf5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e4c51f96c38492db7f269b964f7cf5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46523 } } }
I20260924 00:24:51.076424 19996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:51.076447 19998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:51.076634 20003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:24:51.077203 20003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:24:51.078877 20003 catalog_manager.cc:1347] Generated new cluster ID: b3a0f411223748fabbd92cacc0a861d0
I20260924 00:24:51.078941 20003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:24:51.081535 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:51.077903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50903 (local address 127.0.0.1:46523)
0924 00:24:51.077945 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:51.077947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:51.078062 (+ 115us) server_negotiation.cc:407] Connection header received
0924 00:24:51.078160 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:51.078161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:51.078172 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:51.078201 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:51.078383 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:51.079117 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:51.080074 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:51.080423 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:51.080493 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:51.081223 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:51.081224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:51.081225 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:51.081230 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:51.081246 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:51.081251 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:51.081339 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:51.081397 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 00:24:51.081453 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:24:51.085788 20003 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:24:51.086493 20003 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:24:51.099187 20003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e4c51f96c38492db7f269b964f7cf5a: Generated new TSK 0
I20260924 00:24:51.099437 20003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:24:51.104758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 19923
2026-09-24T00:24:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:24:51 dist-test-slave-777x krb5kdc[20022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:24:51 dist-test-slave-777x krb5kdc[20022](info): set up 2 sockets
Sep 24 00:24:51 dist-test-slave-777x krb5kdc[20022](info): commencing operation
krb5kdc: starting...
W20260924 00:24:52.200558 392 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:24:52 dist-test-slave-777x krb5kdc[20022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:24:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:24:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:24:52.328235 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45289
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:52.417104 20035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:52.417353 20035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:52.417382 20035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:52.418766 20035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:24:52.418818 20035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:52.418834 20035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:52.418848 20035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:24:52.418862 20035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:24:52.421813 20035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:52.422297 20035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:52.422616 20035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:52.428818 20041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:52.428833 20043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:52.428892 20040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:52.429031 20035 server_base.cc:1034] running on GCE node
I20260924 00:24:52.429337 20035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:52.429689 20035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:52.430858 20035 hybrid_clock.cc:648] HybridClock initialized: now 1790209492430851 us; error 34 us; skew 500 ppm
Sep 24 00:24:52 dist-test-slave-777x krb5kdc[20022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:52.444214 20035 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:24:52.444718 20035 webserver.cc:460] Webserver started at http://127.0.0.1:45279/ using document root <none> and password file <none>
I20260924 00:24:52.444962 20035 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:52.445031 20035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:52.445192 20035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:52.446353 20035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "6d41ab83b9004eb9b47f65184a5a18bb"
format_stamp: "Formatted at 2026-09-24 00:24:52 on dist-test-slave-777x"
server_key: "9949dad32a9f0ef229407c14e0b45327"
server_key_iv: "d555ac34237fb94d535ead4a3acd7ee0"
server_key_version: "encryptionkey@0"
I20260924 00:24:52.446732 20035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "6d41ab83b9004eb9b47f65184a5a18bb"
format_stamp: "Formatted at 2026-09-24 00:24:52 on dist-test-slave-777x"
server_key: "9949dad32a9f0ef229407c14e0b45327"
server_key_iv: "d555ac34237fb94d535ead4a3acd7ee0"
server_key_version: "encryptionkey@0"
I20260924 00:24:52.448266 20035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:24:52.449215 20050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:52.449469 20035 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:52.449606 20035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "6d41ab83b9004eb9b47f65184a5a18bb"
format_stamp: "Formatted at 2026-09-24 00:24:52 on dist-test-slave-777x"
server_key: "9949dad32a9f0ef229407c14e0b45327"
server_key_iv: "d555ac34237fb94d535ead4a3acd7ee0"
server_key_version: "encryptionkey@0"
I20260924 00:24:52.449692 20035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:52.453116 20035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:52.455345 20035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:52.455513 20035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:52.459654 20035 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45289
I20260924 00:24:52.459689 20102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45289 every 8 connection(s)
I20260924 00:24:52.460055 20035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:24:52.461290 20103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:52.463551 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 20035
I20260924 00:24:52.463719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:24:52.463938 392 external_mini_cluster.cc:1428] Setting key b363f0f900b524d8036a563eca9e790d
I20260924 00:24:52.464367 20103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb: Bootstrap starting.
I20260924 00:24:52.465116 20103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:52.465427 20103 log.cc:824] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:52.466265 20103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb: No bootstrap required, opened a new log
I20260924 00:24:52.466686 20106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:52.464653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42489 (local address 127.0.0.1:45289)
0924 00:24:52.464812 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:52.464822 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:52.464858 (+ 36us) server_negotiation.cc:407] Connection header received
0924 00:24:52.465342 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:52.465349 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:52.465377 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:52.465494 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:52.466263 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:52.466267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:52.466267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:52.466272 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:52.466338 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:52.466354 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:52.466438 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:52.466474 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 00:24:52.466515 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260924 00:24:52.467861 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:24:52.468861 20103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d41ab83b9004eb9b47f65184a5a18bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45289 } }
I20260924 00:24:52.469079 20103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:52.469144 20103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d41ab83b9004eb9b47f65184a5a18bb, State: Initialized, Role: FOLLOWER
I20260924 00:24:52.469281 20103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d41ab83b9004eb9b47f65184a5a18bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45289 } }
I20260924 00:24:52.469388 20103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:24:52.469434 20103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:24:52.469491 20103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:52.470278 20103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d41ab83b9004eb9b47f65184a5a18bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45289 } }
I20260924 00:24:52.470455 20103 leader_election.cc:302] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d41ab83b9004eb9b47f65184a5a18bb; no voters:
I20260924 00:24:52.470571 20106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:52.469312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60765 (local address 127.0.0.1:45289)
0924 00:24:52.469369 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:52.469371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:52.469521 (+ 150us) server_negotiation.cc:407] Connection header received
0924 00:24:52.469541 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:52.469542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:52.469547 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:52.469592 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:52.470242 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:52.470243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:52.470244 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:52.470247 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:52.470259 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:52.470263 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:52.470306 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:52.470383 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 00:24:52.470397 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 00:24:52.470692 20103 leader_election.cc:288] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:24:52.470741 20113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:52.470880 20113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb [term 1 LEADER]: Becoming Leader. State: Replica: 6d41ab83b9004eb9b47f65184a5a18bb, State: Running, Role: LEADER
I20260924 00:24:52.471005 20113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d41ab83b9004eb9b47f65184a5a18bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45289 } }
I20260924 00:24:52.471012 20103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:24:52.471349 20114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d41ab83b9004eb9b47f65184a5a18bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d41ab83b9004eb9b47f65184a5a18bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45289 } } }
I20260924 00:24:52.471446 20114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:52.471341 20113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d41ab83b9004eb9b47f65184a5a18bb. Latest consensus state: current_term: 1 leader_uuid: "6d41ab83b9004eb9b47f65184a5a18bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d41ab83b9004eb9b47f65184a5a18bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45289 } } }
I20260924 00:24:52.471637 20113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:52.471755 20118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:24:52.472246 20118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:24:52.473803 20118 catalog_manager.cc:1347] Generated new cluster ID: 98595f4fb27a46dc923e7283ba3c3996
I20260924 00:24:52.473858 20118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:24:52.498219 20118 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:24:52.498472 20118 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:24:52.503218 20118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d41ab83b9004eb9b47f65184a5a18bb: Generated new TSK 0
I20260924 00:24:52.503434 20118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:24:52.511269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 20035
2026-09-24T00:24:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:24:52 dist-test-slave-777x krb5kdc[20134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:24:52 dist-test-slave-777x krb5kdc[20134](info): set up 2 sockets
Sep 24 00:24:52 dist-test-slave-777x krb5kdc[20134](info): commencing operation
krb5kdc: starting...
W20260924 00:24:53.552588 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:24:53 dist-test-slave-777x krb5kdc[20134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:24:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:24:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:24:53.678946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34935
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:53.762133 20147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:53.762379 20147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:53.762418 20147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:53.763592 20147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:24:53.763679 20147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:53.763706 20147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:53.763746 20147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:24:53.763769 20147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:24:53.765343 20147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:53.765638 20147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:53.765877 20147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:53.771304 20153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:53.771303 20152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:53.771278 20155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:53.771452 20147 server_base.cc:1034] running on GCE node
I20260924 00:24:53.771742 20147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:53.772079 20147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:53.773265 20147 hybrid_clock.cc:648] HybridClock initialized: now 1790209493773242 us; error 41 us; skew 500 ppm
Sep 24 00:24:53 dist-test-slave-777x krb5kdc[20134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:53.775964 20147 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:24:53.776542 20147 webserver.cc:460] Webserver started at http://127.0.0.1:36767/ using document root <none> and password file <none>
I20260924 00:24:53.776823 20147 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:53.776903 20147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:53.777104 20147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:53.778329 20147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "7cff1a571b0b4d30acdf505f5ef31e84"
format_stamp: "Formatted at 2026-09-24 00:24:53 on dist-test-slave-777x"
server_key: "867f5e7561c0ead50d9a63333960f7ad"
server_key_iv: "3a34dc460745e2317b64fdae9e6b1488"
server_key_version: "encryptionkey@0"
I20260924 00:24:53.778692 20147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "7cff1a571b0b4d30acdf505f5ef31e84"
format_stamp: "Formatted at 2026-09-24 00:24:53 on dist-test-slave-777x"
server_key: "867f5e7561c0ead50d9a63333960f7ad"
server_key_iv: "3a34dc460745e2317b64fdae9e6b1488"
server_key_version: "encryptionkey@0"
I20260924 00:24:53.780120 20147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:24:53.781018 20162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:53.781219 20147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:53.781337 20147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "7cff1a571b0b4d30acdf505f5ef31e84"
format_stamp: "Formatted at 2026-09-24 00:24:53 on dist-test-slave-777x"
server_key: "867f5e7561c0ead50d9a63333960f7ad"
server_key_iv: "3a34dc460745e2317b64fdae9e6b1488"
server_key_version: "encryptionkey@0"
I20260924 00:24:53.781424 20147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:53.801752 20147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:53.804272 20147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:53.804459 20147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:53.809051 20147 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34935
I20260924 00:24:53.809116 20214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34935 every 8 connection(s)
I20260924 00:24:53.809510 20147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:24:53.810528 20215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:53.812830 20215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84: Bootstrap starting.
I20260924 00:24:53.813360 20215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:53.813598 20215 log.cc:824] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:53.814260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 20147
I20260924 00:24:53.814399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:24:53.814440 20215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84: No bootstrap required, opened a new log
I20260924 00:24:53.814606 392 external_mini_cluster.cc:1428] Setting key ac55745f4beac0ff27b04919134add87
I20260924 00:24:53.816828 20215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cff1a571b0b4d30acdf505f5ef31e84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34935 } }
I20260924 00:24:53.816990 20215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:53.817014 20215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cff1a571b0b4d30acdf505f5ef31e84, State: Initialized, Role: FOLLOWER
I20260924 00:24:53.817163 20215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cff1a571b0b4d30acdf505f5ef31e84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34935 } }
I20260924 00:24:53.817227 20215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:24:53.817250 20215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:24:53.817318 20215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:53.818182 20215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cff1a571b0b4d30acdf505f5ef31e84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34935 } }
I20260924 00:24:53.818301 20215 leader_election.cc:302] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cff1a571b0b4d30acdf505f5ef31e84; no voters:
I20260924 00:24:53.818467 20215 leader_election.cc:288] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:24:53.818588 20220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:53.818744 20220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84 [term 1 LEADER]: Becoming Leader. State: Replica: 7cff1a571b0b4d30acdf505f5ef31e84, State: Running, Role: LEADER
I20260924 00:24:53.818755 20215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:24:53.820204 20220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cff1a571b0b4d30acdf505f5ef31e84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34935 } }
I20260924 00:24:53.820855 20222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cff1a571b0b4d30acdf505f5ef31e84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cff1a571b0b4d30acdf505f5ef31e84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34935 } } }
I20260924 00:24:53.821249 20222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84 [sys.catalog]: This master's current role is: LEADER
W20260924 00:24:53.821329 20234 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:24:53.821374 20234 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:24:53.821059 20221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cff1a571b0b4d30acdf505f5ef31e84. Latest consensus state: current_term: 1 leader_uuid: "7cff1a571b0b4d30acdf505f5ef31e84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cff1a571b0b4d30acdf505f5ef31e84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34935 } } }
I20260924 00:24:53.821789 20221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84 [sys.catalog]: This master's current role is: LEADER
Sep 24 00:24:53 dist-test-slave-777x krb5kdc[20134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 00:24:53.823571 20237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:24:53.823787 20237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:24:53.825495 20237 catalog_manager.cc:1347] Generated new cluster ID: 48599a0bcc6b4e2abea18a3a612d02c2
I20260924 00:24:53.825567 20237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:24:53.827817 20219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:53.815277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:51375 (local address 127.0.0.1:34935)
0924 00:24:53.815533 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:53.815543 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:53.815579 (+ 36us) server_negotiation.cc:407] Connection header received
0924 00:24:53.816380 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:53.816390 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:53.816465 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:53.816643 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:53.817516 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:53.819212 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:53.820190 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:53.820816 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:53.820931 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:53.823963 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:53.823968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:53.823968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:53.823988 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:53.826183 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:53.826627 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:53.826631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:53.826631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:53.826724 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:53.826964 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:53.826965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:53.826965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:53.827210 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:53.827385 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:53.827511 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 00:24:53.827613 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":97,"threads_started":1}
I20260924 00:24:53.828815 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:24:53.836951 20237 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:24:53.837579 20219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:53.830252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:60257 (local address 127.0.0.1:34935)
0924 00:24:53.830291 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:53.830293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:53.830301 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:24:53.830324 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:53.830324 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:53.830334 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:53.830389 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:53.830945 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:53.831454 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:53.832159 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:53.832447 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:53.832542 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:53.834239 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:53.834241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:53.834241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:53.834250 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:53.836023 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:53.836477 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:53.836490 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:53.836491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:53.836546 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:53.836791 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:53.836792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:53.836792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:53.836916 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:53.837018 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:53.837437 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 00:24:53.837509 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":21}
I20260924 00:24:53.837742 20237 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:24:53.845296 20237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cff1a571b0b4d30acdf505f5ef31e84: Generated new TSK 0
I20260924 00:24:53.845561 20237 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:24:53.848294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 20147
W20260924 00:24:53.853025 20132 connection.cc:535] client connection to 127.0.0.1:34935 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34935): Connection reset by peer (error 104)
2026-09-24T00:24:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:24:53 dist-test-slave-777x krb5kdc[20246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:24:53 dist-test-slave-777x krb5kdc[20246](info): set up 2 sockets
Sep 24 00:24:53 dist-test-slave-777x krb5kdc[20246](info): commencing operation
krb5kdc: starting...
W20260924 00:24:54.928740 392 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:24:55 dist-test-slave-777x krb5kdc[20246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:24:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:24:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:24:55.056161 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36721
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:55.139215 20259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:55.139485 20259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:55.139531 20259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:55.140827 20259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:24:55.140913 20259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:55.140939 20259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:55.140969 20259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:24:55.140998 20259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:24:55.142597 20259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:55.142896 20259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:55.143158 20259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:55.148541 20264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:55.148545 20265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:55.148562 20267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:55.148823 20259 server_base.cc:1034] running on GCE node
I20260924 00:24:55.149150 20259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:55.149502 20259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:55.150671 20259 hybrid_clock.cc:648] HybridClock initialized: now 1790209495150641 us; error 54 us; skew 500 ppm
Sep 24 00:24:55 dist-test-slave-777x krb5kdc[20246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:55.153631 20259 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:24:55.154124 20259 webserver.cc:460] Webserver started at http://127.0.0.1:44299/ using document root <none> and password file <none>
I20260924 00:24:55.154390 20259 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:55.154457 20259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:55.154635 20259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:55.155737 20259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "5290db4d502e43d998bb1f776b303b13"
format_stamp: "Formatted at 2026-09-24 00:24:55 on dist-test-slave-777x"
server_key: "ee86cb744094c354cb614ae609a8ffad"
server_key_iv: "dac5819712eea126ef900d4e3ce2fe3d"
server_key_version: "encryptionkey@0"
I20260924 00:24:55.156323 20259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "5290db4d502e43d998bb1f776b303b13"
format_stamp: "Formatted at 2026-09-24 00:24:55 on dist-test-slave-777x"
server_key: "ee86cb744094c354cb614ae609a8ffad"
server_key_iv: "dac5819712eea126ef900d4e3ce2fe3d"
server_key_version: "encryptionkey@0"
I20260924 00:24:55.157846 20259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:24:55.158768 20274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:55.158986 20259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:55.159159 20259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "5290db4d502e43d998bb1f776b303b13"
format_stamp: "Formatted at 2026-09-24 00:24:55 on dist-test-slave-777x"
server_key: "ee86cb744094c354cb614ae609a8ffad"
server_key_iv: "dac5819712eea126ef900d4e3ce2fe3d"
server_key_version: "encryptionkey@0"
I20260924 00:24:55.159248 20259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:55.192797 20259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:55.195029 20259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:55.195204 20259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:55.199322 20259 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36721
I20260924 00:24:55.199359 20326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36721 every 8 connection(s)
I20260924 00:24:55.199729 20259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:24:55.200671 20327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:55.201849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 20259
I20260924 00:24:55.201997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:24:55.202219 392 external_mini_cluster.cc:1428] Setting key c4ace15e6abee97ee14b60cc2382d587
I20260924 00:24:55.203440 20327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13: Bootstrap starting.
I20260924 00:24:55.204154 20327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:55.204485 20327 log.cc:824] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:55.205209 20327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13: No bootstrap required, opened a new log
I20260924 00:24:55.207453 20327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5290db4d502e43d998bb1f776b303b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36721 } }
I20260924 00:24:55.207652 20327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:55.207693 20327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5290db4d502e43d998bb1f776b303b13, State: Initialized, Role: FOLLOWER
I20260924 00:24:55.207830 20327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5290db4d502e43d998bb1f776b303b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36721 } }
I20260924 00:24:55.207914 20327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:24:55.207953 20327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:24:55.208016 20327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:55.208395 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:55.202793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:52787 (local address 127.0.0.1:36721)
0924 00:24:55.203058 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:55.203068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:55.203102 (+ 34us) server_negotiation.cc:407] Connection header received
0924 00:24:55.203677 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:55.203698 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:55.203780 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:55.203947 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:55.204101 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:55.205440 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:55.206190 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:55.206840 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:55.206971 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:55.207617 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:55.207621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:55.207621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:55.207826 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:55.207866 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:55.207880 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:55.208015 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:55.208122 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 00:24:55.208152 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":126,"threads_started":1}
I20260924 00:24:55.208725 20327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5290db4d502e43d998bb1f776b303b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36721 } }
I20260924 00:24:55.208825 20327 leader_election.cc:302] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5290db4d502e43d998bb1f776b303b13; no voters:
I20260924 00:24:55.209028 20327 leader_election.cc:288] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:24:55.209044 20332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:55.209162 20332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13 [term 1 LEADER]: Becoming Leader. State: Replica: 5290db4d502e43d998bb1f776b303b13, State: Running, Role: LEADER
I20260924 00:24:55.209290 20327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:24:55.209327 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:24:55.209300 20332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5290db4d502e43d998bb1f776b303b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36721 } }
I20260924 00:24:55.209710 20333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5290db4d502e43d998bb1f776b303b13. Latest consensus state: current_term: 1 leader_uuid: "5290db4d502e43d998bb1f776b303b13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5290db4d502e43d998bb1f776b303b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36721 } } }
I20260924 00:24:55.209789 20333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13 [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:55.209940 20334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5290db4d502e43d998bb1f776b303b13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5290db4d502e43d998bb1f776b303b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36721 } } }
I20260924 00:24:55.210000 20334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13 [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:55.210301 20338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:24:55.210812 20338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:24:55.212287 20338 catalog_manager.cc:1347] Generated new cluster ID: d3fcc207b370425094c508a4a25fe7e0
I20260924 00:24:55.212342 20338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:24:55.214712 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:55.211000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:33787 (local address 127.0.0.1:36721)
0924 00:24:55.211039 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:55.211041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:55.211382 (+ 341us) server_negotiation.cc:407] Connection header received
0924 00:24:55.211750 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:55.211751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:55.211759 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:55.211829 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:55.211978 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:55.212649 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:55.213406 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:55.213640 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:55.213727 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:55.214433 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:55.214434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:55.214434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:55.214439 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:55.214453 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:55.214457 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:55.214533 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:55.214601 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 00:24:55.214618 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:24:55.222522 20338 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:24:55.223266 20338 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:24:55.227763 20338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5290db4d502e43d998bb1f776b303b13: Generated new TSK 0
I20260924 00:24:55.227952 20338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:24:55.236409 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 20259
W20260924 00:24:55.240625 20244 connection.cc:535] client connection to 127.0.0.1:36721 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36721): Connection reset by peer (error 104)
2026-09-24T00:24:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:24:55 dist-test-slave-777x krb5kdc[20358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:24:55 dist-test-slave-777x krb5kdc[20358](info): set up 2 sockets
Sep 24 00:24:55 dist-test-slave-777x krb5kdc[20358](info): commencing operation
krb5kdc: starting...
W20260924 00:24:56.280398 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:24:56 dist-test-slave-777x krb5kdc[20358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:24:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:24:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:24:56.408943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35665
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:56.492494 20371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:56.492729 20371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:56.492753 20371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:56.494112 20371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:24:56.494182 20371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:56.494234 20371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:56.494287 20371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:24:56.494328 20371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:24:56.496124 20371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:56.496445 20371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:56.496709 20371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:56.502471 20376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:56.502499 20379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:56.502475 20377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:56.502835 20371 server_base.cc:1034] running on GCE node
I20260924 00:24:56.503086 20371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:56.503372 20371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:56.504576 20371 hybrid_clock.cc:648] HybridClock initialized: now 1790209496504563 us; error 41 us; skew 500 ppm
Sep 24 00:24:56 dist-test-slave-777x krb5kdc[20358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209496, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:56.507282 20371 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:24:56.507807 20371 webserver.cc:460] Webserver started at http://127.0.0.1:33079/ using document root <none> and password file <none>
I20260924 00:24:56.508040 20371 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:56.508109 20371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:56.508277 20371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:56.509405 20371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "35252d61edce4106bb9ee9dc17961f01"
format_stamp: "Formatted at 2026-09-24 00:24:56 on dist-test-slave-777x"
server_key: "6d6c7b70895db4b3f68de444d09a6222"
server_key_iv: "279a127a4e6d44f0899565164d0bf0f7"
server_key_version: "encryptionkey@0"
I20260924 00:24:56.509791 20371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "35252d61edce4106bb9ee9dc17961f01"
format_stamp: "Formatted at 2026-09-24 00:24:56 on dist-test-slave-777x"
server_key: "6d6c7b70895db4b3f68de444d09a6222"
server_key_iv: "279a127a4e6d44f0899565164d0bf0f7"
server_key_version: "encryptionkey@0"
I20260924 00:24:56.511247 20371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:24:56.512135 20386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:56.512383 20371 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:56.512494 20371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "35252d61edce4106bb9ee9dc17961f01"
format_stamp: "Formatted at 2026-09-24 00:24:56 on dist-test-slave-777x"
server_key: "6d6c7b70895db4b3f68de444d09a6222"
server_key_iv: "279a127a4e6d44f0899565164d0bf0f7"
server_key_version: "encryptionkey@0"
I20260924 00:24:56.512619 20371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:56.516649 20371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:56.519541 20371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:56.519719 20371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:56.524406 20371 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35665
I20260924 00:24:56.524446 20438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35665 every 8 connection(s)
I20260924 00:24:56.524817 20371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:24:56.525841 20439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:56.528249 20439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01: Bootstrap starting.
I20260924 00:24:56.528861 20439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:56.529148 20439 log.cc:824] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:56.529915 20439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01: No bootstrap required, opened a new log
I20260924 00:24:56.531515 20439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35252d61edce4106bb9ee9dc17961f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260924 00:24:56.531718 20439 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:56.531790 20439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35252d61edce4106bb9ee9dc17961f01, State: Initialized, Role: FOLLOWER
I20260924 00:24:56.531975 20439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35252d61edce4106bb9ee9dc17961f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260924 00:24:56.532078 20439 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:24:56.532117 20439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:24:56.532200 20439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:56.532771 20439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35252d61edce4106bb9ee9dc17961f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260924 00:24:56.532892 20439 leader_election.cc:302] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35252d61edce4106bb9ee9dc17961f01; no voters:
I20260924 00:24:56.533083 20439 leader_election.cc:288] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:24:56.533135 20442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:56.533353 20442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01 [term 1 LEADER]: Becoming Leader. State: Replica: 35252d61edce4106bb9ee9dc17961f01, State: Running, Role: LEADER
I20260924 00:24:56.533360 20439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:24:56.533463 20442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35252d61edce4106bb9ee9dc17961f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } }
I20260924 00:24:56.533901 20444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35252d61edce4106bb9ee9dc17961f01. Latest consensus state: current_term: 1 leader_uuid: "35252d61edce4106bb9ee9dc17961f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35252d61edce4106bb9ee9dc17961f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } } }
I20260924 00:24:56.533885 20443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35252d61edce4106bb9ee9dc17961f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35252d61edce4106bb9ee9dc17961f01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35665 } } }
I20260924 00:24:56.534024 20444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01 [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:56.534096 20443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01 [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:56.534351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 20371
I20260924 00:24:56.534482 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:24:56.534646 392 external_mini_cluster.cc:1428] Setting key 4746515aa3779e99dca7ce6efab04808
I20260924 00:24:56.535086 20455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:24:56.535607 20455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:24:56.538059 20460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:56.535316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:50225 (local address 127.0.0.1:35665)
0924 00:24:56.536298 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:56.536308 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:56.536346 (+ 38us) server_negotiation.cc:407] Connection header received
0924 00:24:56.536472 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:56.536480 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:56.536513 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:56.536682 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:56.537464 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:56.537468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:56.537469 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:56.537474 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:56.537521 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:56.537538 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:56.537630 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:56.537762 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 00:24:56.537812 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":919,"thread_start_us":68,"threads_started":1}
I20260924 00:24:56.539386 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:24:56.540684 20455 catalog_manager.cc:1347] Generated new cluster ID: b003a36e715e4b48991e4a84eb44cfb2
I20260924 00:24:56.540763 20455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:24:56.542424 20460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:56.541085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:55985 (local address 127.0.0.1:35665)
0924 00:24:56.541121 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:56.541123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:56.541399 (+ 276us) server_negotiation.cc:407] Connection header received
0924 00:24:56.541451 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:56.541452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:56.541458 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:56.541526 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:56.542201 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:56.542203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:56.542203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:56.542207 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:56.542224 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:56.542229 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:56.542287 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:56.542344 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 00:24:56.542358 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 00:24:56.560689 20455 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:24:56.560904 20455 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:24:56.572204 20455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35252d61edce4106bb9ee9dc17961f01: Generated new TSK 0
I20260924 00:24:56.572460 20455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:24:56.574626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 20371
2026-09-24T00:24:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:24:56 dist-test-slave-777x krb5kdc[20470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:24:56 dist-test-slave-777x krb5kdc[20470](info): set up 2 sockets
Sep 24 00:24:56 dist-test-slave-777x krb5kdc[20470](info): commencing operation
krb5kdc: starting...
W20260924 00:24:57.652520 392 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:24:57 dist-test-slave-777x krb5kdc[20470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:24:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:24:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:24:57.784615 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35991
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:57.873425 20483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:57.873710 20483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:57.873755 20483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:57.875156 20483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:24:57.875258 20483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:57.875280 20483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:57.875301 20483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:24:57.875334 20483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:24:57.877046 20483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35991
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:57.877388 20483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:57.877643 20483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:24:57.883026 20483 server_base.cc:1034] running on GCE node
W20260924 00:24:57.883044 20491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:57.883016 20489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:57.883263 20488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:57.883603 20483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:57.883955 20483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:57.885162 20483 hybrid_clock.cc:648] HybridClock initialized: now 1790209497885093 us; error 80 us; skew 500 ppm
Sep 24 00:24:57 dist-test-slave-777x krb5kdc[20470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:57.893324 20483 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:24:57.893895 20483 webserver.cc:460] Webserver started at http://127.0.0.1:41545/ using document root <none> and password file <none>
I20260924 00:24:57.894182 20483 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:57.894277 20483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:57.894445 20483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:57.895663 20483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "72d6f12163d14cbbb023ee718ba00e15"
format_stamp: "Formatted at 2026-09-24 00:24:57 on dist-test-slave-777x"
server_key: "7b088c1caf73dabd9b8d80f45a74f0e5"
server_key_iv: "85cc8caca94541d3bfcbb73d566faa5d"
server_key_version: "encryptionkey@0"
I20260924 00:24:57.896080 20483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "72d6f12163d14cbbb023ee718ba00e15"
format_stamp: "Formatted at 2026-09-24 00:24:57 on dist-test-slave-777x"
server_key: "7b088c1caf73dabd9b8d80f45a74f0e5"
server_key_iv: "85cc8caca94541d3bfcbb73d566faa5d"
server_key_version: "encryptionkey@0"
I20260924 00:24:57.897742 20483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:24:57.898777 20498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:57.899021 20483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:57.899148 20483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "72d6f12163d14cbbb023ee718ba00e15"
format_stamp: "Formatted at 2026-09-24 00:24:57 on dist-test-slave-777x"
server_key: "7b088c1caf73dabd9b8d80f45a74f0e5"
server_key_iv: "85cc8caca94541d3bfcbb73d566faa5d"
server_key_version: "encryptionkey@0"
I20260924 00:24:57.899326 20483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:57.911356 20483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:57.914235 20483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:57.914445 20483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:57.918566 20483 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35991
I20260924 00:24:57.918604 20550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35991 every 8 connection(s)
I20260924 00:24:57.918999 20483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:24:57.919950 20551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:57.920415 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 20483
I20260924 00:24:57.920563 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:24:57.920758 392 external_mini_cluster.cc:1428] Setting key 5122a6368559f097b1a7aade705edacf
I20260924 00:24:57.923275 20551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15: Bootstrap starting.
I20260924 00:24:57.923961 20551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:57.924367 20551 log.cc:824] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:57.925194 20551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15: No bootstrap required, opened a new log
Sep 24 00:24:57 dist-test-slave-777x krb5kdc[20470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 00:24:57.927807 20551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d6f12163d14cbbb023ee718ba00e15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35991 } }
I20260924 00:24:57.928025 20551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:57.928074 20551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d6f12163d14cbbb023ee718ba00e15, State: Initialized, Role: FOLLOWER
I20260924 00:24:57.928274 20551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d6f12163d14cbbb023ee718ba00e15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35991 } }
I20260924 00:24:57.928377 20551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:24:57.928423 20551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:24:57.928473 20551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:57.929054 20551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d6f12163d14cbbb023ee718ba00e15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35991 } }
I20260924 00:24:57.929174 20551 leader_election.cc:302] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72d6f12163d14cbbb023ee718ba00e15; no voters:
I20260924 00:24:57.929384 20551 leader_election.cc:288] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:24:57.929416 20556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:57.929522 20556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15 [term 1 LEADER]: Becoming Leader. State: Replica: 72d6f12163d14cbbb023ee718ba00e15, State: Running, Role: LEADER
I20260924 00:24:57.929636 20556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d6f12163d14cbbb023ee718ba00e15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35991 } }
I20260924 00:24:57.929665 20551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:24:57.931043 20558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72d6f12163d14cbbb023ee718ba00e15. Latest consensus state: current_term: 1 leader_uuid: "72d6f12163d14cbbb023ee718ba00e15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d6f12163d14cbbb023ee718ba00e15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35991 } } }
I20260924 00:24:57.931223 20557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72d6f12163d14cbbb023ee718ba00e15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72d6f12163d14cbbb023ee718ba00e15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35991 } } }
I20260924 00:24:57.931461 20558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15 [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:57.931519 20557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15 [sys.catalog]: This master's current role is: LEADER
W20260924 00:24:57.931483 20571 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:24:57.931565 20571 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:24:57.931722 20573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:24:57.931888 20573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:24:57.932588 20554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:57.921342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:44087 (local address 127.0.0.1:35991)
0924 00:24:57.921521 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:57.921529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:57.921556 (+ 27us) server_negotiation.cc:407] Connection header received
0924 00:24:57.922048 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:57.922055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:57.922111 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:57.922244 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:57.923017 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:57.924023 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:57.925048 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:57.925543 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:57.925629 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:57.928558 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:57.928563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:57.928563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:57.928578 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:57.930695 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:57.931192 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:57.931195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:57.931195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:57.931315 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:57.931632 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:57.931633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:57.931634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:57.931954 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:57.932115 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:57.932273 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 00:24:57.932384 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260924 00:24:57.933612 20573 catalog_manager.cc:1347] Generated new cluster ID: d5f3707e4f4241b08681145b282272ab
I20260924 00:24:57.933712 20573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:24:57.933919 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:24:57.944010 20554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:57.935432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:56759 (local address 127.0.0.1:35991)
0924 00:24:57.935468 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:57.935470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:57.935762 (+ 292us) server_negotiation.cc:407] Connection header received
0924 00:24:57.935812 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:57.935813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:57.935824 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:57.935889 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:57.936591 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:57.937679 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:57.938807 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:57.939141 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:57.939230 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:57.940858 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:57.940860 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:57.940861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:24:57.940869 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:57.942625 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:57.943088 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:57.943089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:57.943089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:57.943156 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:24:57.943410 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:24:57.943411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:24:57.943412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:24:57.943554 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:57.943666 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:57.943837 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 00:24:57.943908 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 00:24:57.970520 20573 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:24:57.971349 20573 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:24:57.980388 20573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72d6f12163d14cbbb023ee718ba00e15: Generated new TSK 0
I20260924 00:24:57.980651 20573 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:24:57.983774 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 20483
W20260924 00:24:57.988291 20468 connection.cc:535] client connection to 127.0.0.1:35991 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35991): Connection reset by peer (error 104)
2026-09-24T00:24:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1409 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:24:58 dist-test-slave-777x krb5kdc[20582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:24:58 dist-test-slave-777x krb5kdc[20582](info): set up 2 sockets
Sep 24 00:24:58 dist-test-slave-777x krb5kdc[20582](info): commencing operation
krb5kdc: starting...
W20260924 00:24:59.092835 392 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:24:59 dist-test-slave-777x krb5kdc[20582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209499, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:24:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:24:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:24:59.224440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45225
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:24:59.311515 20595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:24:59.311803 20595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:24:59.311842 20595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:24:59.313098 20595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:24:59.313180 20595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:24:59.313202 20595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:24:59.313223 20595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:24:59.313253 20595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:24:59.314931 20595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45225
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:24:59.315237 20595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:24:59.315498 20595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:24:59.320776 20603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:59.320793 20600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:24:59.320824 20601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:24:59.321156 20595 server_base.cc:1034] running on GCE node
I20260924 00:24:59.321410 20595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:24:59.321725 20595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:24:59.322901 20595 hybrid_clock.cc:648] HybridClock initialized: now 1790209499322891 us; error 42 us; skew 500 ppm
Sep 24 00:24:59 dist-test-slave-777x krb5kdc[20582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209499, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:24:59.325762 20595 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:24:59.326289 20595 webserver.cc:460] Webserver started at http://127.0.0.1:40887/ using document root <none> and password file <none>
I20260924 00:24:59.326546 20595 fs_manager.cc:359] Metadata directory not provided
I20260924 00:24:59.326622 20595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:24:59.326774 20595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:24:59.327891 20595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "26188627c18c4428a7cd7d8e62496c19"
format_stamp: "Formatted at 2026-09-24 00:24:59 on dist-test-slave-777x"
server_key: "1d5f3c7c8850db4d538f2f3f2d60239b"
server_key_iv: "6e8c25debfd443cc5b08eac846a2072f"
server_key_version: "encryptionkey@0"
I20260924 00:24:59.328426 20595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "26188627c18c4428a7cd7d8e62496c19"
format_stamp: "Formatted at 2026-09-24 00:24:59 on dist-test-slave-777x"
server_key: "1d5f3c7c8850db4d538f2f3f2d60239b"
server_key_iv: "6e8c25debfd443cc5b08eac846a2072f"
server_key_version: "encryptionkey@0"
I20260924 00:24:59.329874 20595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:24:59.330796 20610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:24:59.330999 20595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:24:59.331120 20595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "26188627c18c4428a7cd7d8e62496c19"
format_stamp: "Formatted at 2026-09-24 00:24:59 on dist-test-slave-777x"
server_key: "1d5f3c7c8850db4d538f2f3f2d60239b"
server_key_iv: "6e8c25debfd443cc5b08eac846a2072f"
server_key_version: "encryptionkey@0"
I20260924 00:24:59.331203 20595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:24:59.365872 20595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:24:59.368307 20595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:24:59.368494 20595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:24:59.372632 20595 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45225
I20260924 00:24:59.372704 20662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45225 every 8 connection(s)
I20260924 00:24:59.373070 20595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:24:59.374156 20663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:24:59.376880 20663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19: Bootstrap starting.
I20260924 00:24:59.377550 20663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19: Neither blocks nor log segments found. Creating new log.
I20260924 00:24:59.377947 20663 log.cc:824] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19: Log is configured to *not* fsync() on all Append() calls
I20260924 00:24:59.378827 20663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19: No bootstrap required, opened a new log
I20260924 00:24:59.380599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 20595
I20260924 00:24:59.380694 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:24:59.380913 392 external_mini_cluster.cc:1428] Setting key 37751656a27af16779a50515074a09b1
I20260924 00:24:59.380967 20663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26188627c18c4428a7cd7d8e62496c19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } }
I20260924 00:24:59.381193 20663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:24:59.381250 20663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26188627c18c4428a7cd7d8e62496c19, State: Initialized, Role: FOLLOWER
I20260924 00:24:59.381433 20663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26188627c18c4428a7cd7d8e62496c19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } }
I20260924 00:24:59.381565 20663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:24:59.381664 20663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:24:59.381737 20663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:24:59.382598 20663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26188627c18c4428a7cd7d8e62496c19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } }
I20260924 00:24:59.382709 20663 leader_election.cc:302] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26188627c18c4428a7cd7d8e62496c19; no voters:
I20260924 00:24:59.382915 20663 leader_election.cc:288] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:24:59.382952 20668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:24:59.383131 20668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19 [term 1 LEADER]: Becoming Leader. State: Replica: 26188627c18c4428a7cd7d8e62496c19, State: Running, Role: LEADER
I20260924 00:24:59.383226 20663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:24:59.383280 20668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26188627c18c4428a7cd7d8e62496c19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } }
I20260924 00:24:59.383618 20668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26188627c18c4428a7cd7d8e62496c19. Latest consensus state: current_term: 1 leader_uuid: "26188627c18c4428a7cd7d8e62496c19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26188627c18c4428a7cd7d8e62496c19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } } }
I20260924 00:24:59.383633 20670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26188627c18c4428a7cd7d8e62496c19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26188627c18c4428a7cd7d8e62496c19" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } } }
I20260924 00:24:59.383816 20670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19 [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:59.383805 20668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19 [sys.catalog]: This master's current role is: LEADER
I20260924 00:24:59.384291 20676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:24:59.384765 20676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:24:59.387060 20676 catalog_manager.cc:1347] Generated new cluster ID: b475abe601564416bf41cc2e5cf9aa4c
I20260924 00:24:59.387193 20676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:24:59.387220 20667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:59.381515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:55415 (local address 127.0.0.1:45225)
0924 00:24:59.381723 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:59.381743 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:59.381788 (+ 45us) server_negotiation.cc:407] Connection header received
0924 00:24:59.381916 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:59.381921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:59.381970 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:59.382086 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:59.382394 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:59.383654 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:59.385067 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:59.385661 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:59.385804 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:59.386572 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:59.386576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:59.386576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:59.386753 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:59.386783 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:59.386794 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:59.386889 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:59.386969 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 00:24:59.386999 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":115,"threads_started":1}
I20260924 00:24:59.388144 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:24:59.392655 20667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:24:59.389531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:46847 (local address 127.0.0.1:45225)
0924 00:24:59.389566 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 00:24:59.389568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:24:59.389782 (+ 214us) server_negotiation.cc:407] Connection header received
0924 00:24:59.389828 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:24:59.389829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:24:59.389850 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:24:59.389905 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:24:59.390041 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:59.390527 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:59.391307 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:24:59.391541 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:24:59.391626 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:24:59.392407 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:24:59.392408 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:24:59.392409 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:24:59.392413 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:24:59.392427 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:24:59.392431 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:24:59.392484 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 00:24:59.392543 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 00:24:59.392555 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 00:24:59.411734 20676 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:24:59.412593 20676 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:24:59.418133 20676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26188627c18c4428a7cd7d8e62496c19: Generated new TSK 0
I20260924 00:24:59.418370 20676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:24:59.425909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 20595
W20260924 00:24:59.430542 20580 connection.cc:535] client connection to 127.0.0.1:45225 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45225): Connection reset by peer (error 104)
2026-09-24T00:24:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1442 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:24:59 dist-test-slave-777x krb5kdc[20694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:24:59 dist-test-slave-777x krb5kdc[20694](info): set up 2 sockets
Sep 24 00:24:59 dist-test-slave-777x krb5kdc[20694](info): commencing operation
krb5kdc: starting...
W20260924 00:25:00.520812 392 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:25:00 dist-test-slave-777x krb5kdc[20694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:25:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:25:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 00:25:00.647801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34469
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:25:00.734712 20707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:25:00.734947 20707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:25:00.734970 20707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:25:00.736115 20707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:25:00.736156 20707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:25:00.736202 20707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:25:00.736214 20707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:25:00.736227 20707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:25:00.737695 20707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34469
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:25:00.737900 20707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:25:00.738077 20707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:25:00.743180 20715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:25:00.743221 20712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:25:00.743161 20707 server_base.cc:1034] running on GCE node
W20260924 00:25:00.743161 20713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:25:00.743652 20707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:25:00.743980 20707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:25:00.745179 20707 hybrid_clock.cc:648] HybridClock initialized: now 1790209500745141 us; error 68 us; skew 500 ppm
Sep 24 00:25:00 dist-test-slave-777x krb5kdc[20694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209500, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:25:00.748072 20707 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 00:25:00.748679 20707 webserver.cc:460] Webserver started at http://127.0.0.1:45579/ using document root <none> and password file <none>
I20260924 00:25:00.748936 20707 fs_manager.cc:359] Metadata directory not provided
I20260924 00:25:00.749011 20707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:25:00.749245 20707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:25:00.750504 20707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "c1aff09b4a0b44a2b1238afa931eefc4"
format_stamp: "Formatted at 2026-09-24 00:25:00 on dist-test-slave-777x"
server_key: "05ed979d388c8f7fe6795a596b1f4e6a"
server_key_iv: "783652369ea17fde347fea18b7865879"
server_key_version: "encryptionkey@0"
I20260924 00:25:00.750931 20707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "c1aff09b4a0b44a2b1238afa931eefc4"
format_stamp: "Formatted at 2026-09-24 00:25:00 on dist-test-slave-777x"
server_key: "05ed979d388c8f7fe6795a596b1f4e6a"
server_key_iv: "783652369ea17fde347fea18b7865879"
server_key_version: "encryptionkey@0"
I20260924 00:25:00.752590 20707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:25:00.753571 20722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:25:00.753885 20707 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:25:00.754021 20707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "c1aff09b4a0b44a2b1238afa931eefc4"
format_stamp: "Formatted at 2026-09-24 00:25:00 on dist-test-slave-777x"
server_key: "05ed979d388c8f7fe6795a596b1f4e6a"
server_key_iv: "783652369ea17fde347fea18b7865879"
server_key_version: "encryptionkey@0"
I20260924 00:25:00.754104 20707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:25:00.757766 20707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:25:00.759990 20707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:25:00.760201 20707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:25:00.764572 20707 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34469
I20260924 00:25:00.764596 20774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34469 every 8 connection(s)
I20260924 00:25:00.764966 20707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:25:00.766000 20775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:25:00.768431 20775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4: Bootstrap starting.
I20260924 00:25:00.769027 20775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4: Neither blocks nor log segments found. Creating new log.
I20260924 00:25:00.769325 20775 log.cc:824] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4: Log is configured to *not* fsync() on all Append() calls
I20260924 00:25:00.770293 20775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4: No bootstrap required, opened a new log
I20260924 00:25:00.772251 20775 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1aff09b4a0b44a2b1238afa931eefc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34469 } }
I20260924 00:25:00.772414 20775 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:25:00.772444 20775 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1aff09b4a0b44a2b1238afa931eefc4, State: Initialized, Role: FOLLOWER
I20260924 00:25:00.772650 20775 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1aff09b4a0b44a2b1238afa931eefc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34469 } }
I20260924 00:25:00.772744 20775 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:25:00.772794 20775 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:25:00.772853 20775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:25:00.773326 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 20707
I20260924 00:25:00.773465 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:25:00.773727 392 external_mini_cluster.cc:1428] Setting key 2fc7bdb712a6a555cc53707341356440
I20260924 00:25:00.773743 20775 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1aff09b4a0b44a2b1238afa931eefc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34469 } }
I20260924 00:25:00.773860 20775 leader_election.cc:302] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1aff09b4a0b44a2b1238afa931eefc4; no voters:
I20260924 00:25:00.774101 20775 leader_election.cc:288] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:25:00.774163 20778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:25:00.774379 20778 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4 [term 1 LEADER]: Becoming Leader. State: Replica: c1aff09b4a0b44a2b1238afa931eefc4, State: Running, Role: LEADER
I20260924 00:25:00.774501 20775 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:25:00.774557 20778 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1aff09b4a0b44a2b1238afa931eefc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34469 } }
I20260924 00:25:00.775123 20781 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1aff09b4a0b44a2b1238afa931eefc4. Latest consensus state: current_term: 1 leader_uuid: "c1aff09b4a0b44a2b1238afa931eefc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1aff09b4a0b44a2b1238afa931eefc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34469 } } }
I20260924 00:25:00.775271 20781 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4 [sys.catalog]: This master's current role is: LEADER
I20260924 00:25:00.775614 20779 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1aff09b4a0b44a2b1238afa931eefc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1aff09b4a0b44a2b1238afa931eefc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34469 } } }
I20260924 00:25:00.775770 20779 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4 [sys.catalog]: This master's current role is: LEADER
I20260924 00:25:00.776437 20787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:25:00.777297 20782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:00.774393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:58937 (local address 127.0.0.1:34469)
0924 00:25:00.775102 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:00.775113 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:00.775219 (+ 106us) server_negotiation.cc:407] Connection header received
0924 00:25:00.775359 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:00.775367 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:00.775403 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:00.775863 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:00.776710 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:00.776714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:00.776714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:25:00.776718 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:00.776758 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:25:00.776770 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:00.776852 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:00.777033 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 00:25:00.777091 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":83,"threads_started":1}
I20260924 00:25:00.777318 20787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:25:00.778407 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:25:00.780669 20787 catalog_manager.cc:1347] Generated new cluster ID: 6aab32284ea041ac85281bf482be740a
I20260924 00:25:00.780738 20787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:25:00.785609 20782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:00.780916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.127:50627 (local address 127.0.0.1:34469)
0924 00:25:00.780957 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:00.780959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:00.784448 (+ 3489us) server_negotiation.cc:407] Connection header received
0924 00:25:00.784540 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:00.784541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:00.784548 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:00.784649 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:00.785358 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:00.785359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:00.785360 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:25:00.785365 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:00.785381 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:25:00.785386 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:00.785440 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:00.785491 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 00:25:00.785509 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:25:00.803063 20787 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:25:00.803284 20787 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:25:00.814389 20787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1aff09b4a0b44a2b1238afa931eefc4: Generated new TSK 0
I20260924 00:25:00.814651 20787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:25:00.820003 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 20707
2026-09-24T00:25:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:25:00 dist-test-slave-777x krb5kdc[20806](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:25:00 dist-test-slave-777x krb5kdc[20806](info): set up 2 sockets
Sep 24 00:25:00 dist-test-slave-777x krb5kdc[20806](info): commencing operation
krb5kdc: starting...
W20260924 00:25:01.880414 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:25:01 dist-test-slave-777x krb5kdc[20806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:25:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:25:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:25:02.013429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40041
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:25:02.095108 20819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:25:02.095396 20819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:25:02.095441 20819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:25:02.097172 20819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:25:02.097225 20819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:25:02.097241 20819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:25:02.097257 20819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:25:02.097271 20819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:25:02.098884 20819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40041
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:25:02.099159 20819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:25:02.099427 20819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:25:02.104419 20824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:25:02.104423 20827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:25:02.104430 20825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:25:02.104763 20819 server_base.cc:1034] running on GCE node
I20260924 00:25:02.105039 20819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:25:02.105356 20819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:25:02.106534 20819 hybrid_clock.cc:648] HybridClock initialized: now 1790209502106530 us; error 39 us; skew 500 ppm
Sep 24 00:25:02 dist-test-slave-777x krb5kdc[20806](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209502, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:25:02.109072 20819 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:25:02.109568 20819 webserver.cc:460] Webserver started at http://127.0.98.62:44395/ using document root <none> and password file <none>
I20260924 00:25:02.109828 20819 fs_manager.cc:359] Metadata directory not provided
I20260924 00:25:02.109897 20819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:25:02.110044 20819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:25:02.111115 20819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "3f70aa28f78a4348afc77503587a17b0"
format_stamp: "Formatted at 2026-09-24 00:25:02 on dist-test-slave-777x"
server_key: "9e90e7749e2a212a3ca626f08ab75e41"
server_key_iv: "e90e3a5cb02823dac15702943b16b5ea"
server_key_version: "encryptionkey@0"
I20260924 00:25:02.111497 20819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "3f70aa28f78a4348afc77503587a17b0"
format_stamp: "Formatted at 2026-09-24 00:25:02 on dist-test-slave-777x"
server_key: "9e90e7749e2a212a3ca626f08ab75e41"
server_key_iv: "e90e3a5cb02823dac15702943b16b5ea"
server_key_version: "encryptionkey@0"
I20260924 00:25:02.112982 20819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 00:25:02.113924 20834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:25:02.114291 20819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:25:02.114413 20819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "3f70aa28f78a4348afc77503587a17b0"
format_stamp: "Formatted at 2026-09-24 00:25:02 on dist-test-slave-777x"
server_key: "9e90e7749e2a212a3ca626f08ab75e41"
server_key_iv: "e90e3a5cb02823dac15702943b16b5ea"
server_key_version: "encryptionkey@0"
I20260924 00:25:02.114498 20819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:25:02.130024 20819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:25:02.132508 20819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:25:02.132701 20819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:25:02.137245 20819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40041
I20260924 00:25:02.137284 20886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40041 every 8 connection(s)
I20260924 00:25:02.137671 20819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:25:02.138545 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 20819
I20260924 00:25:02.138688 20887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:25:02.138718 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:25:02.138962 392 external_mini_cluster.cc:1428] Setting key b4bacd5eb4000b00168c0cdaa09d746b
I20260924 00:25:02.142426 20887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0: Bootstrap starting.
I20260924 00:25:02.143206 20887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0: Neither blocks nor log segments found. Creating new log.
I20260924 00:25:02.143527 20887 log.cc:824] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0: Log is configured to *not* fsync() on all Append() calls
I20260924 00:25:02.144439 20887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0: No bootstrap required, opened a new log
I20260924 00:25:02.146744 20887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f70aa28f78a4348afc77503587a17b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40041 } }
I20260924 00:25:02.147019 20887 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:25:02.147064 20887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f70aa28f78a4348afc77503587a17b0, State: Initialized, Role: FOLLOWER
Sep 24 00:25:02 dist-test-slave-777x krb5kdc[20806](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:25:02.147212 20887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f70aa28f78a4348afc77503587a17b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40041 } }
I20260924 00:25:02.147315 20887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:25:02.147354 20887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:25:02.147408 20887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:25:02.148221 20887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f70aa28f78a4348afc77503587a17b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40041 } }
I20260924 00:25:02.148350 20887 leader_election.cc:302] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f70aa28f78a4348afc77503587a17b0; no voters:
I20260924 00:25:02.148559 20887 leader_election.cc:288] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:25:02.148624 20892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:25:02.148836 20892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0 [term 1 LEADER]: Becoming Leader. State: Replica: 3f70aa28f78a4348afc77503587a17b0, State: Running, Role: LEADER
I20260924 00:25:02.148890 20887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:25:02.148943 20892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f70aa28f78a4348afc77503587a17b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40041 } }
I20260924 00:25:02.149302 20892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f70aa28f78a4348afc77503587a17b0. Latest consensus state: current_term: 1 leader_uuid: "3f70aa28f78a4348afc77503587a17b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f70aa28f78a4348afc77503587a17b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40041 } } }
I20260924 00:25:02.149313 20893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f70aa28f78a4348afc77503587a17b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f70aa28f78a4348afc77503587a17b0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40041 } } }
I20260924 00:25:02.149400 20892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0 [sys.catalog]: This master's current role is: LEADER
I20260924 00:25:02.149433 20893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0 [sys.catalog]: This master's current role is: LEADER
W20260924 00:25:02.150995 20908 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:25:02.151070 20908 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:25:02.151180 20898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:25:02.151453 20898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:25:02.152288 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:02.139568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59017 (local address 127.0.98.62:40041)
0924 00:25:02.139811 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:02.139821 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:02.139859 (+ 38us) server_negotiation.cc:407] Connection header received
0924 00:25:02.140480 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:02.140490 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:02.140562 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:02.140732 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:02.141481 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:02.142913 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:02.144055 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:02.144658 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:02.144807 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:02.147852 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:02.147856 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:02.147857 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:02.147869 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:02.150245 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:02.150790 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:02.150794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:02.150795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:02.150897 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:02.151175 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:02.151177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:02.151177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:02.151536 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:02.151699 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:02.151934 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 00:25:02.152043 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":119,"threads_started":1}
I20260924 00:25:02.153294 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:25:02.153607 20898 catalog_manager.cc:1347] Generated new cluster ID: f4562fa03b0a4c24834ff36300e8fb73
I20260924 00:25:02.153661 20898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:25:02.159883 20898 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:25:02.160859 20898 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:25:02.165210 20898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f70aa28f78a4348afc77503587a17b0: Generated new TSK 0
I20260924 00:25:02.165452 20898 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:25:02.168052 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:02.159600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49205 (local address 127.0.98.62:40041)
0924 00:25:02.159643 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:02.159646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:02.159791 (+ 145us) server_negotiation.cc:407] Connection header received
0924 00:25:02.159898 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:02.159899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:02.159910 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:02.159942 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:02.160743 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:02.161514 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:02.162560 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:02.162902 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:02.162990 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:02.164604 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:02.164606 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:02.164607 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:02.164616 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:02.166482 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:02.166922 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:02.166924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:02.166924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:02.166997 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:02.167358 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:02.167359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:02.167360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:02.167516 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:02.167642 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:02.167860 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 00:25:02.167937 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:25:02.170095 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 20819
W20260924 00:25:02.174446 20804 connection.cc:535] client connection to 127.0.98.62:40041 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:40041): Connection reset by peer (error 104)
2026-09-24T00:25:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:25:02 dist-test-slave-777x krb5kdc[20918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:25:02 dist-test-slave-777x krb5kdc[20918](info): set up 2 sockets
Sep 24 00:25:02 dist-test-slave-777x krb5kdc[20918](info): commencing operation
krb5kdc: starting...
W20260924 00:25:03.256494 392 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:25:03 dist-test-slave-777x krb5kdc[20918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:25:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:25:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:25:03.387630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46061
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:25:03.473517 20931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:25:03.473747 20931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:25:03.473773 20931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:25:03.474920 20931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:25:03.474964 20931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:25:03.474982 20931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:25:03.474995 20931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:25:03.475009 20931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:25:03.476492 20931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46061
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:25:03.476702 20931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:25:03.476884 20931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:25:03.482201 20937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:25:03.482262 20936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:25:03.482268 20931 server_base.cc:1034] running on GCE node
W20260924 00:25:03.482221 20939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:25:03.482795 20931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:25:03.483156 20931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:25:03.484335 20931 hybrid_clock.cc:648] HybridClock initialized: now 1790209503484310 us; error 38 us; skew 500 ppm
Sep 24 00:25:03 dist-test-slave-777x krb5kdc[20918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:25:03.491952 20931 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:25:03.492527 20931 webserver.cc:460] Webserver started at http://127.0.98.62:41681/ using document root <none> and password file <none>
I20260924 00:25:03.492805 20931 fs_manager.cc:359] Metadata directory not provided
I20260924 00:25:03.492857 20931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:25:03.492959 20931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:25:03.494099 20931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "9463ea13d16e4f328e18ceb1850f697f"
format_stamp: "Formatted at 2026-09-24 00:25:03 on dist-test-slave-777x"
server_key: "ba183b9637e019a6dec5b5223d695c05"
server_key_iv: "f9eb4c6d28d8822b369e2cb56cca6f4a"
server_key_version: "encryptionkey@0"
I20260924 00:25:03.494427 20931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "9463ea13d16e4f328e18ceb1850f697f"
format_stamp: "Formatted at 2026-09-24 00:25:03 on dist-test-slave-777x"
server_key: "ba183b9637e019a6dec5b5223d695c05"
server_key_iv: "f9eb4c6d28d8822b369e2cb56cca6f4a"
server_key_version: "encryptionkey@0"
I20260924 00:25:03.495841 20931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:25:03.496712 20946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:25:03.496923 20931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 00:25:03.497021 20931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "9463ea13d16e4f328e18ceb1850f697f"
format_stamp: "Formatted at 2026-09-24 00:25:03 on dist-test-slave-777x"
server_key: "ba183b9637e019a6dec5b5223d695c05"
server_key_iv: "f9eb4c6d28d8822b369e2cb56cca6f4a"
server_key_version: "encryptionkey@0"
I20260924 00:25:03.497136 20931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:25:03.522941 20931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:25:03.525734 20931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:25:03.525874 20931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:25:03.530619 20931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46061
I20260924 00:25:03.530658 20998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46061 every 8 connection(s)
I20260924 00:25:03.531083 20931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:25:03.532052 20999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:25:03.533363 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 20931
I20260924 00:25:03.533529 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:25:03.533776 392 external_mini_cluster.cc:1428] Setting key 903211bc1dca338cf4ef9f081743762f
I20260924 00:25:03.534878 20999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f: Bootstrap starting.
I20260924 00:25:03.535540 20999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f: Neither blocks nor log segments found. Creating new log.
I20260924 00:25:03.535848 20999 log.cc:824] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f: Log is configured to *not* fsync() on all Append() calls
I20260924 00:25:03.536639 20999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f: No bootstrap required, opened a new log
I20260924 00:25:03.539033 20999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9463ea13d16e4f328e18ceb1850f697f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46061 } }
I20260924 00:25:03.539255 20999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:25:03.539304 20999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9463ea13d16e4f328e18ceb1850f697f, State: Initialized, Role: FOLLOWER
I20260924 00:25:03.539460 20999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9463ea13d16e4f328e18ceb1850f697f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46061 } }
I20260924 00:25:03.539542 20999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:25:03.539565 20999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:25:03.539616 20999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:25:03.540288 21002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:03.534404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43047 (local address 127.0.98.62:46061)
0924 00:25:03.534584 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:03.534593 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:03.534622 (+ 29us) server_negotiation.cc:407] Connection header received
0924 00:25:03.535174 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:03.535182 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:03.535238 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:03.535387 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:03.535691 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:03.536846 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:03.537952 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:03.538435 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:03.538539 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:03.539393 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:03.539398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:03.539398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:25:03.539670 (+ 272us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:03.539715 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:25:03.539730 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:03.539827 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:03.539988 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 00:25:03.540025 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260924 00:25:03.540369 20999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9463ea13d16e4f328e18ceb1850f697f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46061 } }
I20260924 00:25:03.540482 20999 leader_election.cc:302] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9463ea13d16e4f328e18ceb1850f697f; no voters:
I20260924 00:25:03.540664 20999 leader_election.cc:288] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:25:03.540691 21004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:25:03.540802 21004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f [term 1 LEADER]: Becoming Leader. State: Replica: 9463ea13d16e4f328e18ceb1850f697f, State: Running, Role: LEADER
I20260924 00:25:03.540930 20999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:25:03.540943 21004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9463ea13d16e4f328e18ceb1850f697f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46061 } }
I20260924 00:25:03.541308 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:25:03.541359 21005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9463ea13d16e4f328e18ceb1850f697f. Latest consensus state: current_term: 1 leader_uuid: "9463ea13d16e4f328e18ceb1850f697f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9463ea13d16e4f328e18ceb1850f697f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46061 } } }
I20260924 00:25:03.541445 21005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f [sys.catalog]: This master's current role is: LEADER
I20260924 00:25:03.541574 21006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9463ea13d16e4f328e18ceb1850f697f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9463ea13d16e4f328e18ceb1850f697f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46061 } } }
I20260924 00:25:03.541716 21006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f [sys.catalog]: This master's current role is: LEADER
W20260924 00:25:03.542856 21020 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:25:03.543155 21020 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:25:03.543313 21013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:25:03.543530 21013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:25:03.545785 21013 catalog_manager.cc:1347] Generated new cluster ID: 08e682ffbc35463998c18237ae489e5a
I20260924 00:25:03.545851 21013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:25:03.546850 21002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:03.542922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48081 (local address 127.0.98.62:46061)
0924 00:25:03.542962 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:03.542965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:03.543318 (+ 353us) server_negotiation.cc:407] Connection header received
0924 00:25:03.543393 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:03.543394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:03.543405 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:03.543451 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:03.543631 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:03.544141 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:03.545382 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:03.545662 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:03.545728 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:03.546581 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:03.546583 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:03.546583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:25:03.546588 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:03.546603 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:25:03.546607 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:03.546659 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:03.546748 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 00:25:03.546758 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 00:25:03.558408 21013 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:25:03.559017 21013 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:25:03.567409 21013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9463ea13d16e4f328e18ceb1850f697f: Generated new TSK 0
I20260924 00:25:03.567694 21013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:25:03.575273 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 20931
W20260924 00:25:03.579535 20916 connection.cc:535] client connection to 127.0.98.62:46061 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:46061): Connection reset by peer (error 104)
2026-09-24T00:25:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:25:03 dist-test-slave-777x krb5kdc[21030](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:25:03 dist-test-slave-777x krb5kdc[21030](info): set up 2 sockets
Sep 24 00:25:03 dist-test-slave-777x krb5kdc[21030](info): commencing operation
krb5kdc: starting...
W20260924 00:25:04.676370 392 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:25:04 dist-test-slave-777x krb5kdc[21030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:25:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:25:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:25:04.807996 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37281
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:25:04.900893 21043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:25:04.901122 21043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:25:04.901146 21043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:25:04.902323 21043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:25:04.902365 21043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:25:04.902379 21043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:25:04.902391 21043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:25:04.902403 21043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:25:04.903964 21043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37281
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:25:04.904246 21043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:25:04.904497 21043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:25:04.909763 21043 server_base.cc:1034] running on GCE node
W20260924 00:25:04.909715 21049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:25:04.909734 21048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:25:04.909721 21051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:25:04.910156 21043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:25:04.910457 21043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:25:04.911654 21043 hybrid_clock.cc:648] HybridClock initialized: now 1790209504911633 us; error 61 us; skew 500 ppm
Sep 24 00:25:04 dist-test-slave-777x krb5kdc[21030](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:25:04.914618 21043 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:25:04.915093 21043 webserver.cc:460] Webserver started at http://127.0.98.62:42655/ using document root <none> and password file <none>
I20260924 00:25:04.915340 21043 fs_manager.cc:359] Metadata directory not provided
I20260924 00:25:04.915413 21043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:25:04.915581 21043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:25:04.916827 21043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "1618b3786a9d4175bd2bd9f1a70c5a23"
format_stamp: "Formatted at 2026-09-24 00:25:04 on dist-test-slave-777x"
server_key: "89afdfcb53f08cbad80871610b08ac8a"
server_key_iv: "d16d411775f519380e8eb063bb24f7df"
server_key_version: "encryptionkey@0"
I20260924 00:25:04.917263 21043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "1618b3786a9d4175bd2bd9f1a70c5a23"
format_stamp: "Formatted at 2026-09-24 00:25:04 on dist-test-slave-777x"
server_key: "89afdfcb53f08cbad80871610b08ac8a"
server_key_iv: "d16d411775f519380e8eb063bb24f7df"
server_key_version: "encryptionkey@0"
I20260924 00:25:04.918790 21043 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:25:04.919755 21058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:25:04.919965 21043 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:25:04.920079 21043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "1618b3786a9d4175bd2bd9f1a70c5a23"
format_stamp: "Formatted at 2026-09-24 00:25:04 on dist-test-slave-777x"
server_key: "89afdfcb53f08cbad80871610b08ac8a"
server_key_iv: "d16d411775f519380e8eb063bb24f7df"
server_key_version: "encryptionkey@0"
I20260924 00:25:04.920280 21043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:25:04.923979 21043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:25:04.926337 21043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:25:04.926528 21043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:25:04.930824 21043 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37281
I20260924 00:25:04.930856 21110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37281 every 8 connection(s)
I20260924 00:25:04.931201 21043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:25:04.932248 21111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:25:04.933295 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 21043
I20260924 00:25:04.933430 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:25:04.933615 392 external_mini_cluster.cc:1428] Setting key a385f5e179daa690f2225b4b212286a0
I20260924 00:25:04.934850 21111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23: Bootstrap starting.
I20260924 00:25:04.935472 21111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23: Neither blocks nor log segments found. Creating new log.
I20260924 00:25:04.935791 21111 log.cc:824] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23: Log is configured to *not* fsync() on all Append() calls
I20260924 00:25:04.936298 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:04.934177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46159 (local address 127.0.98.62:37281)
0924 00:25:04.934409 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:04.934419 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:04.934452 (+ 33us) server_negotiation.cc:407] Connection header received
0924 00:25:04.934944 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:04.934954 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:04.934991 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:04.935148 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:04.935846 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:04.935850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:04.935851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:25:04.935855 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:04.935899 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:25:04.935914 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:04.935994 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:04.936033 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 00:25:04.936073 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":88,"threads_started":1}
I20260924 00:25:04.936635 21111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23: No bootstrap required, opened a new log
I20260924 00:25:04.937253 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:25:04.939445 21111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1618b3786a9d4175bd2bd9f1a70c5a23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37281 } }
I20260924 00:25:04.939647 21111 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:25:04.939687 21111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1618b3786a9d4175bd2bd9f1a70c5a23, State: Initialized, Role: FOLLOWER
I20260924 00:25:04.939822 21111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1618b3786a9d4175bd2bd9f1a70c5a23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37281 } }
I20260924 00:25:04.939924 21111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:25:04.939973 21111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:25:04.940032 21111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:25:04.940421 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:04.939175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57431 (local address 127.0.98.62:37281)
0924 00:25:04.939222 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:04.939224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:04.939356 (+ 132us) server_negotiation.cc:407] Connection header received
0924 00:25:04.939384 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:04.939385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:04.939390 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:04.939457 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:04.940053 (+ 596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:04.940054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:04.940054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:25:04.940058 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:04.940074 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:25:04.940078 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:04.940145 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:04.940351 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 00:25:04.940365 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 00:25:04.940841 21111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1618b3786a9d4175bd2bd9f1a70c5a23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37281 } }
I20260924 00:25:04.940955 21111 leader_election.cc:302] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1618b3786a9d4175bd2bd9f1a70c5a23; no voters:
I20260924 00:25:04.941113 21111 leader_election.cc:288] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:25:04.941174 21121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:25:04.941352 21121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23 [term 1 LEADER]: Becoming Leader. State: Replica: 1618b3786a9d4175bd2bd9f1a70c5a23, State: Running, Role: LEADER
I20260924 00:25:04.941371 21111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:25:04.941442 21121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1618b3786a9d4175bd2bd9f1a70c5a23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37281 } }
I20260924 00:25:04.941797 21122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1618b3786a9d4175bd2bd9f1a70c5a23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1618b3786a9d4175bd2bd9f1a70c5a23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37281 } } }
I20260924 00:25:04.941901 21122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23 [sys.catalog]: This master's current role is: LEADER
I20260924 00:25:04.941802 21123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1618b3786a9d4175bd2bd9f1a70c5a23. Latest consensus state: current_term: 1 leader_uuid: "1618b3786a9d4175bd2bd9f1a70c5a23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1618b3786a9d4175bd2bd9f1a70c5a23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37281 } } }
I20260924 00:25:04.942157 21123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23 [sys.catalog]: This master's current role is: LEADER
I20260924 00:25:04.942452 21130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:25:04.942979 21130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:25:04.944622 21130 catalog_manager.cc:1347] Generated new cluster ID: 1d478dceab584611bfafbb2b8d9d457d
I20260924 00:25:04.944669 21130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:25:04.950559 21130 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:25:04.950740 21130 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:25:04.959456 21130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1618b3786a9d4175bd2bd9f1a70c5a23: Generated new TSK 0
I20260924 00:25:04.959686 21130 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:25:04.967360 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 21043
2026-09-24T00:25:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1392 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:25:05 dist-test-slave-777x krb5kdc[21142](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:25:05 dist-test-slave-777x krb5kdc[21142](info): set up 2 sockets
Sep 24 00:25:05 dist-test-slave-777x krb5kdc[21142](info): commencing operation
krb5kdc: starting...
W20260924 00:25:06.008411 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:25:06 dist-test-slave-777x krb5kdc[21142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:25:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:25:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:25:06.139227 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40463
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:25:06.225227 21155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:25:06.225478 21155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:25:06.225518 21155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:25:06.226753 21155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:25:06.226825 21155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:25:06.226852 21155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:25:06.226883 21155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:25:06.226905 21155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:25:06.228719 21155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:25:06.229019 21155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:25:06.229279 21155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:25:06.234644 21155 server_base.cc:1034] running on GCE node
W20260924 00:25:06.234608 21160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:25:06.234627 21163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:25:06.234628 21161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:25:06.235152 21155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:25:06.235466 21155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:25:06.236646 21155 hybrid_clock.cc:648] HybridClock initialized: now 1790209506236640 us; error 39 us; skew 500 ppm
Sep 24 00:25:06 dist-test-slave-777x krb5kdc[21142](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209506, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:25:06.239181 21155 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:25:06.239678 21155 webserver.cc:460] Webserver started at http://127.0.98.62:45135/ using document root <none> and password file <none>
I20260924 00:25:06.239907 21155 fs_manager.cc:359] Metadata directory not provided
I20260924 00:25:06.239954 21155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:25:06.240093 21155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:25:06.241173 21155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "1f3ec4df6e004397bba3a85d904ea13b"
format_stamp: "Formatted at 2026-09-24 00:25:06 on dist-test-slave-777x"
server_key: "697c02929e1ca07d6e9e243010a5abc9"
server_key_iv: "defe68a78c28e22412e42c447763ec51"
server_key_version: "encryptionkey@0"
I20260924 00:25:06.241536 21155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "1f3ec4df6e004397bba3a85d904ea13b"
format_stamp: "Formatted at 2026-09-24 00:25:06 on dist-test-slave-777x"
server_key: "697c02929e1ca07d6e9e243010a5abc9"
server_key_iv: "defe68a78c28e22412e42c447763ec51"
server_key_version: "encryptionkey@0"
I20260924 00:25:06.242933 21155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:25:06.243731 21170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:25:06.243918 21155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:25:06.244027 21155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "1f3ec4df6e004397bba3a85d904ea13b"
format_stamp: "Formatted at 2026-09-24 00:25:06 on dist-test-slave-777x"
server_key: "697c02929e1ca07d6e9e243010a5abc9"
server_key_iv: "defe68a78c28e22412e42c447763ec51"
server_key_version: "encryptionkey@0"
I20260924 00:25:06.244103 21155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:25:06.262353 21155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:25:06.264552 21155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:25:06.264729 21155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:25:06.268972 21155 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40463
I20260924 00:25:06.269024 21222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40463 every 8 connection(s)
I20260924 00:25:06.269410 21155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:25:06.270424 21223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:25:06.273265 21223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b: Bootstrap starting.
I20260924 00:25:06.273957 21223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b: Neither blocks nor log segments found. Creating new log.
I20260924 00:25:06.274283 21223 log.cc:824] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b: Log is configured to *not* fsync() on all Append() calls
I20260924 00:25:06.274904 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 21155
I20260924 00:25:06.275027 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:25:06.275245 392 external_mini_cluster.cc:1428] Setting key 435628b8b4368a5744b40e1a3a8f81e3
I20260924 00:25:06.275275 21223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b: No bootstrap required, opened a new log
I20260924 00:25:06.277772 21223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3ec4df6e004397bba3a85d904ea13b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40463 } }
I20260924 00:25:06.277989 21223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:25:06.278038 21223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f3ec4df6e004397bba3a85d904ea13b, State: Initialized, Role: FOLLOWER
I20260924 00:25:06.278185 21223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3ec4df6e004397bba3a85d904ea13b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40463 } }
I20260924 00:25:06.278402 21223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:25:06.278441 21223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:25:06.278481 21223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:25:06.279287 21223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3ec4df6e004397bba3a85d904ea13b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40463 } }
I20260924 00:25:06.279397 21223 leader_election.cc:302] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f3ec4df6e004397bba3a85d904ea13b; no voters:
I20260924 00:25:06.279548 21223 leader_election.cc:288] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:25:06.279654 21228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:25:06.279881 21228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b [term 1 LEADER]: Becoming Leader. State: Replica: 1f3ec4df6e004397bba3a85d904ea13b, State: Running, Role: LEADER
I20260924 00:25:06.279953 21223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:25:06.280062 21228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3ec4df6e004397bba3a85d904ea13b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40463 } }
I20260924 00:25:06.280455 21230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f3ec4df6e004397bba3a85d904ea13b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3ec4df6e004397bba3a85d904ea13b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40463 } } }
I20260924 00:25:06.280546 21230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b [sys.catalog]: This master's current role is: LEADER
I20260924 00:25:06.280447 21228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f3ec4df6e004397bba3a85d904ea13b. Latest consensus state: current_term: 1 leader_uuid: "1f3ec4df6e004397bba3a85d904ea13b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f3ec4df6e004397bba3a85d904ea13b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40463 } } }
I20260924 00:25:06.280673 21228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b [sys.catalog]: This master's current role is: LEADER
I20260924 00:25:06.280895 21234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:25:06.281597 21234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:25:06.283313 21234 catalog_manager.cc:1347] Generated new cluster ID: aca052269ea54b73bc673bdff4455614
I20260924 00:25:06.283370 21234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 00:25:06 dist-test-slave-777x krb5kdc[21142](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:25:06.289742 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:06.275838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59523 (local address 127.0.98.62:40463)
0924 00:25:06.276069 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:06.276081 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:06.276133 (+ 52us) server_negotiation.cc:407] Connection header received
0924 00:25:06.276971 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:06.276982 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:06.277052 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:06.277222 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:06.277869 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:06.279682 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:06.281412 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:06.282199 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:06.282387 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:06.285622 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:06.285628 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:06.285628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:06.285649 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:06.287896 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:06.288311 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:06.288315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:06.288316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:06.288454 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:06.288674 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:06.288674 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:06.288675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:06.289045 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:06.289247 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:06.289349 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 00:25:06.289506 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":105,"threads_started":1}
I20260924 00:25:06.290889 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:25:06.300204 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:06.292393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40273 (local address 127.0.98.62:40463)
0924 00:25:06.292437 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:06.292440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:06.292625 (+ 185us) server_negotiation.cc:407] Connection header received
0924 00:25:06.292678 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:06.292679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:06.292691 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:06.292763 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:06.293354 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:06.294215 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:06.295205 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:06.295584 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:06.295668 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:06.297164 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:06.297166 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:06.297166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:06.297174 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:06.298929 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:06.299321 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:06.299322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:06.299323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:06.299389 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:06.299592 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:06.299593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:06.299593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:06.299740 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:06.299848 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:06.299949 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 00:25:06.300032 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 00:25:06.301994 21234 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:25:06.302841 21234 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:25:06.306607 21234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f3ec4df6e004397bba3a85d904ea13b: Generated new TSK 0
I20260924 00:25:06.306820 21234 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:25:06.310362 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 21155
2026-09-24T00:25:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:25:06 dist-test-slave-777x krb5kdc[21254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:25:06 dist-test-slave-777x krb5kdc[21254](info): set up 2 sockets
Sep 24 00:25:06 dist-test-slave-777x krb5kdc[21254](info): commencing operation
krb5kdc: starting...
W20260924 00:25:07.445173 392 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:25:07 dist-test-slave-777x krb5kdc[21254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:25:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:25:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:25:07.577450 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37517
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:25:07.663836 21267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:25:07.664083 21267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:25:07.664109 21267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:25:07.665385 21267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:25:07.665432 21267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:25:07.665446 21267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:25:07.665460 21267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:25:07.665472 21267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:25:07.667063 21267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:25:07.667339 21267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:25:07.667593 21267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:25:07.673017 21275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:25:07.673017 21272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:25:07.673058 21273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:25:07.673043 21267 server_base.cc:1034] running on GCE node
I20260924 00:25:07.673528 21267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:25:07.673843 21267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:25:07.675022 21267 hybrid_clock.cc:648] HybridClock initialized: now 1790209507674995 us; error 44 us; skew 500 ppm
Sep 24 00:25:07 dist-test-slave-777x krb5kdc[21254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209507, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:25:07.677886 21267 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:25:07.678462 21267 webserver.cc:460] Webserver started at http://127.0.98.62:40889/ using document root <none> and password file <none>
I20260924 00:25:07.678719 21267 fs_manager.cc:359] Metadata directory not provided
I20260924 00:25:07.678795 21267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:25:07.678944 21267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:25:07.680033 21267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "1f464566e0f24574bbed23c791ede95a"
format_stamp: "Formatted at 2026-09-24 00:25:07 on dist-test-slave-777x"
server_key: "e91db9d249087d597d9752dbe3c04a98"
server_key_iv: "8564e0247bae4d66d72dbea9e88c1f2c"
server_key_version: "encryptionkey@0"
I20260924 00:25:07.680466 21267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "1f464566e0f24574bbed23c791ede95a"
format_stamp: "Formatted at 2026-09-24 00:25:07 on dist-test-slave-777x"
server_key: "e91db9d249087d597d9752dbe3c04a98"
server_key_iv: "8564e0247bae4d66d72dbea9e88c1f2c"
server_key_version: "encryptionkey@0"
I20260924 00:25:07.681972 21267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:25:07.683020 21282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:25:07.683290 21267 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:25:07.683444 21267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "1f464566e0f24574bbed23c791ede95a"
format_stamp: "Formatted at 2026-09-24 00:25:07 on dist-test-slave-777x"
server_key: "e91db9d249087d597d9752dbe3c04a98"
server_key_iv: "8564e0247bae4d66d72dbea9e88c1f2c"
server_key_version: "encryptionkey@0"
I20260924 00:25:07.683534 21267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:25:07.700675 21267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:25:07.703285 21267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:25:07.703478 21267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:25:07.708318 21267 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37517
I20260924 00:25:07.708362 21334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37517 every 8 connection(s)
I20260924 00:25:07.708751 21267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:25:07.709707 21335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:25:07.712105 21335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a: Bootstrap starting.
I20260924 00:25:07.712766 21335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a: Neither blocks nor log segments found. Creating new log.
I20260924 00:25:07.713011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 21267
I20260924 00:25:07.713125 21335 log.cc:824] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a: Log is configured to *not* fsync() on all Append() calls
I20260924 00:25:07.713131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:25:07.713438 392 external_mini_cluster.cc:1428] Setting key c33793f86322577357bd78f1c9ea60b2
I20260924 00:25:07.714152 21335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a: No bootstrap required, opened a new log
I20260924 00:25:07.717432 21335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f464566e0f24574bbed23c791ede95a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260924 00:25:07.717585 21335 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:25:07.717610 21335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f464566e0f24574bbed23c791ede95a, State: Initialized, Role: FOLLOWER
I20260924 00:25:07.717765 21335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f464566e0f24574bbed23c791ede95a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260924 00:25:07.717849 21335 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:25:07.717895 21335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:25:07.717950 21335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:25:07.718772 21335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f464566e0f24574bbed23c791ede95a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260924 00:25:07.718895 21335 leader_election.cc:302] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f464566e0f24574bbed23c791ede95a; no voters:
I20260924 00:25:07.719089 21335 leader_election.cc:288] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:25:07.719151 21340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:25:07.719329 21340 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a [term 1 LEADER]: Becoming Leader. State: Replica: 1f464566e0f24574bbed23c791ede95a, State: Running, Role: LEADER
I20260924 00:25:07.719377 21335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:25:07.719449 21340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f464566e0f24574bbed23c791ede95a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } }
I20260924 00:25:07.719889 21342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f464566e0f24574bbed23c791ede95a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f464566e0f24574bbed23c791ede95a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } } }
I20260924 00:25:07.720001 21342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a [sys.catalog]: This master's current role is: LEADER
I20260924 00:25:07.720264 21341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f464566e0f24574bbed23c791ede95a. Latest consensus state: current_term: 1 leader_uuid: "1f464566e0f24574bbed23c791ede95a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f464566e0f24574bbed23c791ede95a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37517 } } }
I20260924 00:25:07.720386 21341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a [sys.catalog]: This master's current role is: LEADER
I20260924 00:25:07.720280 21347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:25:07.720794 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:07.714012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53553 (local address 127.0.98.62:37517)
0924 00:25:07.714235 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:07.714246 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:07.714289 (+ 43us) server_negotiation.cc:407] Connection header received
0924 00:25:07.714841 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:07.714850 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:07.714921 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:07.715089 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:07.715355 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:07.717089 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:07.718165 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:07.718867 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:07.718990 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:07.719859 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:07.719864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:07.719865 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:25:07.720114 (+ 249us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:07.720189 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:25:07.720220 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:07.720298 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:07.720446 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 00:25:07.720522 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
I20260924 00:25:07.721554 21347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:25:07.722082 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:25:07.723305 21347 catalog_manager.cc:1347] Generated new cluster ID: 6c2b36beff344a4faef08fb5ba2c51ed
I20260924 00:25:07.723359 21347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:25:07.731407 21339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:07.724703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43037 (local address 127.0.98.62:37517)
0924 00:25:07.724809 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:07.724811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:07.728367 (+ 3556us) server_negotiation.cc:407] Connection header received
0924 00:25:07.728450 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:07.728451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:07.728462 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:07.728519 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:07.728679 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:07.729205 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:07.730010 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:07.730246 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:07.730328 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:07.731109 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:07.731110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:07.731111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:25:07.731116 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:07.731130 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:25:07.731135 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:07.731194 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:07.731271 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 00:25:07.731293 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 00:25:07.737473 21347 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:25:07.738070 21347 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:25:07.746332 21347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f464566e0f24574bbed23c791ede95a: Generated new TSK 0
I20260924 00:25:07.746567 21347 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:25:07.754302 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 21267
2026-09-24T00:25:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:25:07 dist-test-slave-777x krb5kdc[21366](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:25:07 dist-test-slave-777x krb5kdc[21366](info): set up 2 sockets
Sep 24 00:25:07 dist-test-slave-777x krb5kdc[21366](info): commencing operation
krb5kdc: starting...
W20260924 00:25:08.808379 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:25:08 dist-test-slave-777x krb5kdc[21366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:25:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:25:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:25:08.940073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44579
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:25:09.022545 21379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:25:09.022778 21379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:25:09.022804 21379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:25:09.024086 21379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:25:09.024140 21379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:25:09.024155 21379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:25:09.024168 21379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:25:09.024248 21379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:25:09.025847 21379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:25:09.026129 21379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:25:09.026379 21379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:25:09.031591 21385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:25:09.031746 21379 server_base.cc:1034] running on GCE node
W20260924 00:25:09.031632 21384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:25:09.031666 21387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:25:09.032094 21379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:25:09.032488 21379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:25:09.033661 21379 hybrid_clock.cc:648] HybridClock initialized: now 1790209509033666 us; error 47 us; skew 500 ppm
Sep 24 00:25:09 dist-test-slave-777x krb5kdc[21366](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:25:09.042589 21379 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:25:09.043136 21379 webserver.cc:460] Webserver started at http://127.0.98.62:43223/ using document root <none> and password file <none>
I20260924 00:25:09.043394 21379 fs_manager.cc:359] Metadata directory not provided
I20260924 00:25:09.043463 21379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:25:09.043605 21379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:25:09.044764 21379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "963aa71f094149fb98df53cc2163e789"
format_stamp: "Formatted at 2026-09-24 00:25:09 on dist-test-slave-777x"
server_key: "a5427c0389f2a7b1b8b82f36ceb26d39"
server_key_iv: "d4a68187d2fcbffd2f0577e772f416a3"
server_key_version: "encryptionkey@0"
I20260924 00:25:09.045131 21379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "963aa71f094149fb98df53cc2163e789"
format_stamp: "Formatted at 2026-09-24 00:25:09 on dist-test-slave-777x"
server_key: "a5427c0389f2a7b1b8b82f36ceb26d39"
server_key_iv: "d4a68187d2fcbffd2f0577e772f416a3"
server_key_version: "encryptionkey@0"
I20260924 00:25:09.046588 21379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:25:09.047523 21394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:25:09.047798 21379 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:25:09.047953 21379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "963aa71f094149fb98df53cc2163e789"
format_stamp: "Formatted at 2026-09-24 00:25:09 on dist-test-slave-777x"
server_key: "a5427c0389f2a7b1b8b82f36ceb26d39"
server_key_iv: "d4a68187d2fcbffd2f0577e772f416a3"
server_key_version: "encryptionkey@0"
I20260924 00:25:09.048035 21379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:25:09.051440 21379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:25:09.053983 21379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:25:09.054140 21379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:25:09.058625 21379 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44579
I20260924 00:25:09.058702 21446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44579 every 8 connection(s)
I20260924 00:25:09.059103 21379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:25:09.060213 21447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:25:09.062649 21447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789: Bootstrap starting.
I20260924 00:25:09.063212 21447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789: Neither blocks nor log segments found. Creating new log.
I20260924 00:25:09.063462 21447 log.cc:824] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789: Log is configured to *not* fsync() on all Append() calls
I20260924 00:25:09.064137 21447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789: No bootstrap required, opened a new log
I20260924 00:25:09.065091 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 21379
I20260924 00:25:09.065232 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:25:09.065433 392 external_mini_cluster.cc:1428] Setting key 8f685629a3d88d9b9292051ce4984713
I20260924 00:25:09.066543 21447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963aa71f094149fb98df53cc2163e789" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44579 } }
I20260924 00:25:09.066748 21447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 00:25:09.066793 21447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 963aa71f094149fb98df53cc2163e789, State: Initialized, Role: FOLLOWER
I20260924 00:25:09.066951 21447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963aa71f094149fb98df53cc2163e789" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44579 } }
I20260924 00:25:09.067034 21447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 00:25:09.067057 21447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 00:25:09.067126 21447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:25:09.068002 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:09.065981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35161 (local address 127.0.98.62:44579)
0924 00:25:09.066235 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:09.066243 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:09.066270 (+ 27us) server_negotiation.cc:407] Connection header received
0924 00:25:09.066502 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:09.066507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:09.066532 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:09.066640 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:09.067448 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:09.067452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:09.067452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:25:09.067457 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:09.067516 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:25:09.067530 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:09.067600 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:09.067702 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 00:25:09.067775 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
I20260924 00:25:09.068348 21447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963aa71f094149fb98df53cc2163e789" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44579 } }
I20260924 00:25:09.068463 21447 leader_election.cc:302] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 963aa71f094149fb98df53cc2163e789; no voters:
I20260924 00:25:09.068672 21447 leader_election.cc:288] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 00:25:09.068689 21452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:25:09.068817 21452 raft_consensus.cc:686] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789 [term 1 LEADER]: Becoming Leader. State: Replica: 963aa71f094149fb98df53cc2163e789, State: Running, Role: LEADER
I20260924 00:25:09.068907 21452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963aa71f094149fb98df53cc2163e789" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44579 } }
I20260924 00:25:09.068967 21447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:25:09.069074 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:25:09.069249 21452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 963aa71f094149fb98df53cc2163e789. Latest consensus state: current_term: 1 leader_uuid: "963aa71f094149fb98df53cc2163e789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963aa71f094149fb98df53cc2163e789" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44579 } } }
I20260924 00:25:09.069321 21452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789 [sys.catalog]: This master's current role is: LEADER
I20260924 00:25:09.069538 21453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "963aa71f094149fb98df53cc2163e789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "963aa71f094149fb98df53cc2163e789" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44579 } } }
I20260924 00:25:09.069636 21453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789 [sys.catalog]: This master's current role is: LEADER
W20260924 00:25:09.070997 21472 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:25:09.071087 21472 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:25:09.071293 21456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:25:09.071481 21456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:25:09.072202 21451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:09.070819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58277 (local address 127.0.98.62:44579)
0924 00:25:09.070860 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:09.070862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:09.071039 (+ 177us) server_negotiation.cc:407] Connection header received
0924 00:25:09.071096 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:09.071097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:09.071104 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:09.071182 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:09.071894 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:09.071895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:09.071895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 00:25:09.071900 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:09.071914 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 00:25:09.071919 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:09.071952 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:09.072070 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 00:25:09.072084 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:25:09.073434 21456 catalog_manager.cc:1347] Generated new cluster ID: 59ac8b62e43744de809e8614303e24b4
I20260924 00:25:09.073500 21456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:25:09.088740 21456 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:25:09.088964 21456 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:25:09.100611 21456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 963aa71f094149fb98df53cc2163e789: Generated new TSK 0
I20260924 00:25:09.100873 21456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:25:09.106508 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 21379
2026-09-24T00:25:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1352 ms)
[----------] 18 tests from AuthTokenIssuingTest (24964 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:25:09 dist-test-slave-777x krb5kdc[21478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:25:09 dist-test-slave-777x krb5kdc[21478](info): set up 2 sockets
Sep 24 00:25:09 dist-test-slave-777x krb5kdc[21478](info): commencing operation
krb5kdc: starting...
W20260924 00:25:10.148483 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:25:10 dist-test-slave-777x krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:25:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:25:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:25:10.277148 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37507
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37507,127.0.98.61:43921,127.0.98.60:35131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:25:10.360337 21491 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 00:25:10.360610 21491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:25:10.360641 21491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:25:10.360687 21491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:25:10.361863 21491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:25:10.361943 21491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:25:10.361966 21491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:25:10.361987 21491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:25:10.362018 21491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:25:10.363550 21491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37507,127.0.98.61:43921,127.0.98.60:35131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:25:10.363843 21491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:25:10.364097 21491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:25:10.369648 21497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:25:10.369616 21496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:25:10.369617 21499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:25:10.369906 21491 server_base.cc:1034] running on GCE node
I20260924 00:25:10.370220 21491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:25:10.370599 21491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:25:10.371810 21491 hybrid_clock.cc:648] HybridClock initialized: now 1790209510371798 us; error 52 us; skew 500 ppm
Sep 24 00:25:10 dist-test-slave-777x krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:25:10.374818 21491 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:25:10.375362 21491 webserver.cc:460] Webserver started at http://127.0.98.62:44879/ using document root <none> and password file <none>
I20260924 00:25:10.375645 21491 fs_manager.cc:359] Metadata directory not provided
I20260924 00:25:10.375725 21491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:25:10.375896 21491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:25:10.377228 21491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58"
format_stamp: "Formatted at 2026-09-24 00:25:10 on dist-test-slave-777x"
server_key: "cc6fa72163813c25c17ba8c0877ed97c"
server_key_iv: "ab165b1b6df877df483f93cb898c7e29"
server_key_version: "encryptionkey@0"
I20260924 00:25:10.377635 21491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58"
format_stamp: "Formatted at 2026-09-24 00:25:10 on dist-test-slave-777x"
server_key: "cc6fa72163813c25c17ba8c0877ed97c"
server_key_iv: "ab165b1b6df877df483f93cb898c7e29"
server_key_version: "encryptionkey@0"
I20260924 00:25:10.379122 21491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:25:10.379966 21506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:25:10.380250 21491 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:25:10.380374 21491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58"
format_stamp: "Formatted at 2026-09-24 00:25:10 on dist-test-slave-777x"
server_key: "cc6fa72163813c25c17ba8c0877ed97c"
server_key_iv: "ab165b1b6df877df483f93cb898c7e29"
server_key_version: "encryptionkey@0"
I20260924 00:25:10.380466 21491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:25:10.391710 21491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:25:10.394328 21491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:25:10.394497 21491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:25:10.398479 21491 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37507
I20260924 00:25:10.398535 21558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37507 every 8 connection(s)
I20260924 00:25:10.398926 21491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:25:10.400012 21559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:25:10.401795 21559 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:25:10.402377 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 21491
I20260924 00:25:10.402469 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:25:10.402715 392 external_mini_cluster.cc:1428] Setting key e6458d0b49ab160feb5182eaad54f356
Sep 24 00:25:10 dist-test-slave-777x krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260924 00:25:10.412788 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.402613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.0.98.62:37507)
0924 00:25:10.402797 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.402808 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.402904 (+ 96us) server_negotiation.cc:407] Connection header received
0924 00:25:10.403490 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.403498 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.403514 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.403606 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.404212 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.405170 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.406310 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.406845 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.406969 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.409412 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:10.409421 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:10.409421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:10.409430 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:10.411140 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.411564 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.411567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.411567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.411657 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.411898 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.411898 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.411899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.412208 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:10.412312 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:10.412512 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 00:25:10.412617 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
I20260924 00:25:10.412801 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.402566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:50320)
0924 00:25:10.402776 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.402818 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:25:10.403323 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:25:10.403623 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:25:10.403626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:25:10.403924 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:25:10.404095 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:10.404099 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.405299 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.405301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:10.406209 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:10.406212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.406966 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.406967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:10.407042 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.407712 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:25:10.407722 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:25:10.409311 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:25:10.411243 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:10.411251 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:10.411253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:10.411452 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:10.411745 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:10.411746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:10.411746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:10.411844 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:10.412305 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:25:10.412312 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:25:10.412387 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 00:25:10.412469 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 00:25:10.412723 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":103,"thread_start_us":125,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260924 00:25:10.413666 21559 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:25:10.414042 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.413903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43921 (local address 127.0.0.1:44150)
0924 00:25:10.413957 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.414009 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 00:25:10.414197 21508 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43921: connect: Connection refused (error 111)
I20260924 00:25:10.414692 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.414589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43921 (local address 127.0.0.1:44166)
0924 00:25:10.414634 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.414663 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 00:25:10.414968 21559 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43921: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43921: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 00:25:10.431821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43921
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37507,127.0.98.61:43921,127.0.98.60:35131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:25:10.464253 21559 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } attempt: 1
I20260924 00:25:10.464864 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.464717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43921 (local address 127.0.0.1:44178)
0924 00:25:10.464787 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.464835 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 00:25:10.465270 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.465178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43921 (local address 127.0.0.1:44186)
0924 00:25:10.465221 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.465252 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 00:25:10.465417 21559 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43921: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43921: connect: Connection refused (error 111)
W20260924 00:25:10.514314 21566 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 00:25:10.514606 21566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:25:10.514639 21566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:25:10.514684 21566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:25:10.515973 21566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:25:10.516053 21566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:25:10.516077 21566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:25:10.516099 21566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:25:10.516127 21566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:25:10.517784 21566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37507,127.0.98.61:43921,127.0.98.60:35131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:25:10.518095 21566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:25:10.518342 21566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:25:10.524107 21566 server_base.cc:1034] running on GCE node
W20260924 00:25:10.524060 21572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:25:10.524065 21571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:25:10.524070 21574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:25:10.524677 21566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:25:10.525048 21566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:25:10.526259 21566 hybrid_clock.cc:648] HybridClock initialized: now 1790209510526211 us; error 63 us; skew 500 ppm
Sep 24 00:25:10 dist-test-slave-777x krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:25:10.529181 21566 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 00:25:10.529647 21566 webserver.cc:460] Webserver started at http://127.0.98.61:43431/ using document root <none> and password file <none>
I20260924 00:25:10.529886 21566 fs_manager.cc:359] Metadata directory not provided
I20260924 00:25:10.529965 21566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:25:10.530124 21566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:25:10.531260 21566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-1/data/instance:
uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e"
format_stamp: "Formatted at 2026-09-24 00:25:10 on dist-test-slave-777x"
server_key: "1300541d323eece7edacfa75cfbe4fcb"
server_key_iv: "30600bdee8bc6144b2a666dadcf995f6"
server_key_version: "encryptionkey@0"
I20260924 00:25:10.531637 21566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-1/wal/instance:
uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e"
format_stamp: "Formatted at 2026-09-24 00:25:10 on dist-test-slave-777x"
server_key: "1300541d323eece7edacfa75cfbe4fcb"
server_key_iv: "30600bdee8bc6144b2a666dadcf995f6"
server_key_version: "encryptionkey@0"
I20260924 00:25:10.533162 21566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:25:10.533556 21559 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } attempt: 2
I20260924 00:25:10.534049 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.533913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43921 (local address 127.0.0.1:44194)
0924 00:25:10.533980 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.534021 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 00:25:10.534243 21581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:25:10.534474 21566 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:25:10.534583 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.534405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43921 (local address 127.0.0.1:44200)
0924 00:25:10.534480 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.534550 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 00:25:10.534581 21566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-1/wal
uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e"
format_stamp: "Formatted at 2026-09-24 00:25:10 on dist-test-slave-777x"
server_key: "1300541d323eece7edacfa75cfbe4fcb"
server_key_iv: "30600bdee8bc6144b2a666dadcf995f6"
server_key_version: "encryptionkey@0"
W20260924 00:25:10.534669 21559 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43921: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43921: connect: Connection refused (error 111)
I20260924 00:25:10.534653 21566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:25:10.549587 21566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:25:10.552052 21566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:25:10.552246 21566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:25:10.556143 21566 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43921
I20260924 00:25:10.556298 21633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43921 every 8 connection(s)
I20260924 00:25:10.556721 21566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-1/data/info.pb
I20260924 00:25:10.557176 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 21566
I20260924 00:25:10.557269 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-1/wal/instance
I20260924 00:25:10.557487 392 external_mini_cluster.cc:1428] Setting key 392a7e371814c6cdc786d05fe59465e1
I20260924 00:25:10.557955 21634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:25:10.559729 21634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 00:25:10 dist-test-slave-777x krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:25:10.574957 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.560568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50332 (local address 127.0.98.62:37507)
0924 00:25:10.560640 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.560643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.560834 (+ 191us) server_negotiation.cc:407] Connection header received
0924 00:25:10.561330 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.561332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.561347 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.561427 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.562096 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.562924 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.564011 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.564385 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.564465 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.571494 (+ 7029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:10.571496 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:10.571497 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:10.571507 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:10.573266 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.573803 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.573804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.573805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.573872 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.574121 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.574122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.574123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.574280 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:10.574350 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:10.574684 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 00:25:10.574800 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 00:25:10.574977 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.560526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:50332)
0924 00:25:10.560703 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.560747 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 00:25:10.561227 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:25:10.561492 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:25:10.561495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:25:10.561786 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:25:10.561994 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:10.561998 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.563049 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.563050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:10.563907 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:10.563909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.564517 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.564518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:10.564687 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.565344 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:25:10.565357 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:25:10.571340 (+ 5983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:25:10.573368 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:10.573379 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:10.573382 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:10.573702 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:10.573944 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:10.573945 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:10.573945 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:10.574057 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:10.574376 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:25:10.574383 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:25:10.574528 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 00:25:10.574629 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 00:25:10.574786 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260924 00:25:10.575682 21634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:25:10 dist-test-slave-777x krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:25:10.585301 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.575979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43921 (local address 127.0.0.1:44214)
0924 00:25:10.576215 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.576226 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:25:10.576272 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:25:10.576410 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:25:10.576411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:25:10.576560 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:25:10.576614 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:10.576616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.577516 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.577518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:10.578045 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:10.578046 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.578575 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.578576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:10.578642 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.579081 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:25:10.579086 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:25:10.581858 (+ 2772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:25:10.583967 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:10.583970 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:10.583971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:10.584245 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:10.584598 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:10.584599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:10.584599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:10.584656 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:10.584911 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:25:10.584912 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:25:10.584953 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 00:25:10.585020 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 00:25:10.585159 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 00:25:10.585462 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.576061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44214 (local address 127.0.98.61:43921)
0924 00:25:10.576250 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.576255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.576277 (+ 22us) server_negotiation.cc:407] Connection header received
0924 00:25:10.576333 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.576336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.576349 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.576412 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.576690 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.577434 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.578142 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.578474 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.578577 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.581966 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:10.581975 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:10.581975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:10.581982 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:10.583807 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.584452 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.584457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.584457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.584526 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.584695 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.584695 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.584696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.584851 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:10.584916 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:10.585094 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 00:25:10.585218 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":159,"thread_start_us":103,"threads_started":1}
I20260924 00:25:10.586143 21634 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:25:10.586544 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.586420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35131 (local address 127.0.0.1:33522)
0924 00:25:10.586472 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.586521 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 00:25:10.586695 21585 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35131: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 00:25:10.587783 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.587628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35131 (local address 127.0.0.1:33532)
0924 00:25:10.587685 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.587751 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 00:25:10.587836 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35131
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37507,127.0.98.61:43921,127.0.98.60:35131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:25:10.588088 21634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35131: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35131: connect: Connection refused (error 111)
I20260924 00:25:10.625808 21559 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } attempt: 3
Sep 24 00:25:10 dist-test-slave-777x krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:25:10.634135 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.626436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44222 (local address 127.0.98.61:43921)
0924 00:25:10.626507 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.626509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.626545 (+ 36us) server_negotiation.cc:407] Connection header received
0924 00:25:10.626589 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.626589 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.626600 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.626670 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.627051 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.627569 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.628443 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.628674 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.628785 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.631202 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:10.631204 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:10.631204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:10.631221 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:10.632856 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.633252 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.633253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.633254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.633320 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.633546 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.633547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.633547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.633675 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:10.633758 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:10.633932 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 00:25:10.634020 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 00:25:10.634135 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.626287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43921 (local address 127.0.0.1:44222)
0924 00:25:10.626379 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.626395 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 00:25:10.626493 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:25:10.626697 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:25:10.626699 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:25:10.626882 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:25:10.626956 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:10.626957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.627679 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.627681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:10.628329 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:10.628332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.628779 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.628780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:10.628864 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.629503 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:25:10.629509 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:25:10.631085 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:25:10.632969 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:10.632970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:10.632971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:10.633165 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:10.633397 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:10.633397 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:10.633398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:10.633467 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:10.633753 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:25:10.633754 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:25:10.633808 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 00:25:10.633905 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 00:25:10.634024 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 00:25:10.634581 21559 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:25:10.634994 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.634859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35131 (local address 127.0.0.1:33548)
0924 00:25:10.634919 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.634961 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 00:25:10.635368 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.635264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35131 (local address 127.0.0.1:33560)
0924 00:25:10.635319 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.635348 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 00:25:10.635511 21559 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35131: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35131: connect: Connection refused (error 111)
I20260924 00:25:10.637373 21634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } attempt: 1
I20260924 00:25:10.637780 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.637656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35131 (local address 127.0.0.1:33562)
0924 00:25:10.637712 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.637742 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 00:25:10.638114 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.638045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35131 (local address 127.0.0.1:33568)
0924 00:25:10.638078 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.638100 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 00:25:10.638229 21634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35131: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35131: connect: Connection refused (error 111)
I20260924 00:25:10.666697 21559 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } attempt: 1
I20260924 00:25:10.667299 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.667124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35131 (local address 127.0.0.1:33576)
0924 00:25:10.667201 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.667253 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 00:25:10.667737 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.667632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35131 (local address 127.0.0.1:33582)
0924 00:25:10.667686 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.667716 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 00:25:10.667902 21559 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35131: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35131: connect: Connection refused (error 111)
W20260924 00:25:10.681962 21641 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 00:25:10.682260 21641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:25:10.682317 21641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:25:10.682358 21641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:25:10.683833 21641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:25:10.683941 21641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:25:10.683971 21641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:25:10.684007 21641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:25:10.684036 21641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:25:10.686038 21641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37507,127.0.98.61:43921,127.0.98.60:35131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:25:10.686374 21641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:25:10.686651 21641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:25:10.692288 21646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:25:10.692464 21641 server_base.cc:1034] running on GCE node
W20260924 00:25:10.692284 21649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:25:10.692258 21647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:25:10.692938 21641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:25:10.693301 21641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:25:10.694482 21641 hybrid_clock.cc:648] HybridClock initialized: now 1790209510694454 us; error 42 us; skew 500 ppm
Sep 24 00:25:10 dist-test-slave-777x krb5kdc[21478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:25:10.697305 21641 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 00:25:10.697763 21641 webserver.cc:460] Webserver started at http://127.0.98.60:41351/ using document root <none> and password file <none>
I20260924 00:25:10.698001 21641 fs_manager.cc:359] Metadata directory not provided
I20260924 00:25:10.698048 21641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:25:10.698195 21641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:25:10.699313 21641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-2/data/instance:
uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb"
format_stamp: "Formatted at 2026-09-24 00:25:10 on dist-test-slave-777x"
server_key: "1341f01fd7866ed8b494ec3e97ca214c"
server_key_iv: "9c57187e5cc4219c24f2b47a1ea9d8b9"
server_key_version: "encryptionkey@0"
I20260924 00:25:10.699718 21641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-2/wal/instance:
uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb"
format_stamp: "Formatted at 2026-09-24 00:25:10 on dist-test-slave-777x"
server_key: "1341f01fd7866ed8b494ec3e97ca214c"
server_key_iv: "9c57187e5cc4219c24f2b47a1ea9d8b9"
server_key_version: "encryptionkey@0"
I20260924 00:25:10.701193 21641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:25:10.702049 21656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:25:10.702263 21641 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 00:25:10.702333 21641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-2/wal
uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb"
format_stamp: "Formatted at 2026-09-24 00:25:10 on dist-test-slave-777x"
server_key: "1341f01fd7866ed8b494ec3e97ca214c"
server_key_iv: "9c57187e5cc4219c24f2b47a1ea9d8b9"
server_key_version: "encryptionkey@0"
I20260924 00:25:10.702435 21641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:25:10.706374 21634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } attempt: 2
I20260924 00:25:10.706833 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.706673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35131 (local address 127.0.0.1:33598)
0924 00:25:10.706736 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.706777 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 00:25:10.707204 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.707101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35131 (local address 127.0.0.1:33604)
0924 00:25:10.707150 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.707186 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 00:25:10.707311 21634 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35131: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35131: connect: Connection refused (error 111)
I20260924 00:25:10.713531 21641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:25:10.715473 21641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:25:10.715653 21641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:25:10.719832 21641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35131
I20260924 00:25:10.719913 21708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35131 every 8 connection(s)
I20260924 00:25:10.720304 21641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-2/data/info.pb
I20260924 00:25:10.721375 21709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:25:10.723043 21709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:25:10.723467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 21641
I20260924 00:25:10.723583 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790209154318147-392-0/minicluster-data/master-2/wal/instance
I20260924 00:25:10.723824 392 external_mini_cluster.cc:1428] Setting key 396bda35fdac44f29ebec614bde00b66
Sep 24 00:25:10 dist-test-slave-777x krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 00:25:10 dist-test-slave-777x krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 00:25:10.734180 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.723915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50340 (local address 127.0.98.62:37507)
0924 00:25:10.723976 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.723978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.724267 (+ 289us) server_negotiation.cc:407] Connection header received
0924 00:25:10.724986 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.724987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.724999 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.725060 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.725627 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.726138 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.727446 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.727745 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.727807 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.730849 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:10.730851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:10.730851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:10.730860 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:10.732640 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.733109 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.733110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.733110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.733175 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.733517 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.733518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.733518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.733651 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:10.733756 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:10.733969 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 00:25:10.734058 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 00:25:10.734197 21713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.724261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33620 (local address 127.0.98.60:35131)
0924 00:25:10.724409 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.724414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.724450 (+ 36us) server_negotiation.cc:407] Connection header received
0924 00:25:10.724852 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.724862 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.724891 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.724988 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.725728 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.726916 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.727758 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.728288 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.728357 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.730815 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:10.730822 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:10.730823 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:10.730830 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:10.732662 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.733027 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.733030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.733030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.733117 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.733330 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.733331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.733331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.733592 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:10.733751 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:10.733871 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 00:25:10.734014 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260924 00:25:10.734413 21711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.723861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37507 (local address 127.0.0.1:50340)
0924 00:25:10.724070 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.724110 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 00:25:10.724849 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:25:10.725085 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:25:10.725092 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:25:10.725366 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:25:10.725529 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:10.725534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.726222 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.726223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:10.727359 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:10.727362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.727840 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.727841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:10.727981 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.728681 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:25:10.728690 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:25:10.730750 (+ 2060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:25:10.732788 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:10.732795 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:10.732798 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:10.733009 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:10.733312 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:10.733312 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:10.733313 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:10.733418 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:10.733727 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:25:10.733734 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:25:10.733815 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 00:25:10.733911 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 00:25:10.734177 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":287,"thread_start_us":114,"threads_started":1}
I20260924 00:25:10.734931 21709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:25:10 dist-test-slave-777x krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 00:25:10 dist-test-slave-777x krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:25:10.742419 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.735283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44234 (local address 127.0.98.61:43921)
0924 00:25:10.735335 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.735337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.735377 (+ 40us) server_negotiation.cc:407] Connection header received
0924 00:25:10.735454 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.735455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.735463 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.735503 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.735831 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.736367 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.737099 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.737310 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.737386 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.739629 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:10.739630 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:10.739631 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:10.739638 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:10.741279 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.741632 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.741634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.741634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.741697 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.741895 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.741896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.741896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.742019 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:10.742095 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:10.742248 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 00:25:10.742318 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 00:25:10.742424 21711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.735200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43921 (local address 127.0.0.1:44234)
0924 00:25:10.735333 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.735344 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:25:10.735391 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:25:10.735509 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:25:10.735510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:25:10.735672 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:25:10.735754 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:10.735756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.736448 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.736449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:10.737016 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:10.737018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.737419 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.737420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:10.737490 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.737972 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:25:10.737977 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:25:10.739538 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:25:10.741373 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:10.741374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:10.741374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:10.741544 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:10.741763 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:10.741764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:10.741765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:10.741823 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:10.742087 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:25:10.742088 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:25:10.742132 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 00:25:10.742221 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 00:25:10.742318 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260924 00:25:10.742789 21709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:25:10.743041 21559 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } attempt: 2
I20260924 00:25:10.744042 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.735079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50352 (local address 127.0.98.62:37507)
0924 00:25:10.735117 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.735119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.735141 (+ 22us) server_negotiation.cc:407] Connection header received
0924 00:25:10.735282 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.735283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.735294 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.735368 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.736067 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.736648 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.737450 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.737729 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.737804 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.740268 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:10.740270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:10.740270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:10.740278 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:10.741823 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.742197 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.742199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.742199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.742273 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.742875 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.742876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.742876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.743250 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:10.743428 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:10.743829 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 00:25:10.743932 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 00:25:10 dist-test-slave-777x krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 00:25:10 dist-test-slave-777x krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 00:25:10 dist-test-slave-777x krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:25:10.751875 21711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.743104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35131 (local address 127.0.0.1:33636)
0924 00:25:10.743486 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.743499 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 00:25:10.743553 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:25:10.743725 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:25:10.743725 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:25:10.743869 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:25:10.743913 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:10.743915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.744941 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.744942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:10.745692 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:10.745693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.746209 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.746210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:10.746568 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.746997 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:25:10.747002 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:25:10.748542 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:25:10.750473 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:10.750474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:10.750475 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:10.750744 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:10.750938 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:10.750939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:10.750939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:10.751016 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:10.751453 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:25:10.751453 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:25:10.751514 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 00:25:10.751587 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 00:25:10.751689 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361}
I20260924 00:25:10.752215 21713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.743152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33636 (local address 127.0.98.60:35131)
0924 00:25:10.743193 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.743195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.743561 (+ 366us) server_negotiation.cc:407] Connection header received
0924 00:25:10.743608 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.743608 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.743636 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.743689 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.743981 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.744817 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.745780 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.746133 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.746210 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.748636 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:10.748637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:10.748638 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:10.748644 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:10.750420 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.750825 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.750826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.750827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.750880 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.751073 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.751074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.751074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.751233 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:10.751334 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:10.752026 (+ 692us) server_negotiation.cc:299] Negotiation successful
0924 00:25:10.752101 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:25:10.752368 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.743269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35131 (local address 127.0.0.1:33644)
0924 00:25:10.743309 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.743320 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:25:10.743369 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:25:10.743655 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:25:10.743656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:25:10.743827 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:25:10.743871 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:10.743873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.744920 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.744921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:10.745700 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:10.745703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.746080 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.746081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:10.746136 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.746572 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:25:10.746577 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:25:10.747964 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:25:10.751002 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:10.751003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:10.751003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:10.751233 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:10.751455 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:10.751456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:10.751457 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:10.751527 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:10.751827 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:25:10.751829 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:25:10.751882 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 00:25:10.751965 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 00:25:10.752082 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 00:25:10.752838 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.743284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33644 (local address 127.0.98.60:35131)
0924 00:25:10.743518 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.743521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.743528 (+ 7us) server_negotiation.cc:407] Connection header received
0924 00:25:10.743552 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.743553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.743565 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.743642 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.743969 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.744823 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.745789 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.746005 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.746081 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.748060 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:10.748062 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:10.748062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:10.748070 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:10.750918 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.751318 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.751319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.751319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.751376 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.751603 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.751604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.751605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.751738 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:10.751811 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:10.752597 (+ 786us) server_negotiation.cc:299] Negotiation successful
0924 00:25:10.752685 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":154,"threads_started":1}
I20260924 00:25:10.754084 21709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55c1e1a2ed8c4222873f2d1497e1e0fb: Bootstrap starting.
I20260924 00:25:10.754156 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.744830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44238 (local address 127.0.98.61:43921)
0924 00:25:10.744871 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.744874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.744882 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:25:10.744930 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.744932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.744942 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.744994 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.745524 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.746237 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.747086 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.747313 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.747398 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.750441 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:10.750443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:10.750444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:10.750452 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:10.752573 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.753152 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.753154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.753154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.753221 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.753471 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.753472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.753473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.753611 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:10.753790 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:10.753843 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 00:25:10.753928 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:25:10.754879 21559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58: Bootstrap starting.
I20260924 00:25:10.754983 21709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55c1e1a2ed8c4222873f2d1497e1e0fb: Neither blocks nor log segments found. Creating new log.
I20260924 00:25:10.755298 21709 log.cc:824] T 00000000000000000000000000000000 P 55c1e1a2ed8c4222873f2d1497e1e0fb: Log is configured to *not* fsync() on all Append() calls
I20260924 00:25:10.755577 21559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58: Neither blocks nor log segments found. Creating new log.
I20260924 00:25:10.755810 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:25:10.755934 21559 log.cc:824] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58: Log is configured to *not* fsync() on all Append() calls
I20260924 00:25:10.756412 21709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55c1e1a2ed8c4222873f2d1497e1e0fb: No bootstrap required, opened a new log
I20260924 00:25:10.756748 21559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58: No bootstrap required, opened a new log
I20260924 00:25:10.756920 21533 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50352
I20260924 00:25:10.758646 21709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55c1e1a2ed8c4222873f2d1497e1e0fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } } peers { permanent_uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } }
I20260924 00:25:10.758796 21709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55c1e1a2ed8c4222873f2d1497e1e0fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55c1e1a2ed8c4222873f2d1497e1e0fb, State: Initialized, Role: FOLLOWER
I20260924 00:25:10.758783 21559 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } } peers { permanent_uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } }
I20260924 00:25:10.758890 21709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55c1e1a2ed8c4222873f2d1497e1e0fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } } peers { permanent_uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } }
I20260924 00:25:10.758945 21559 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce3aa9c0b0f34b628935a58bd1ca5e58, State: Initialized, Role: FOLLOWER
I20260924 00:25:10.759021 21559 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } } peers { permanent_uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } }
I20260924 00:25:10.759212 21709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55c1e1a2ed8c4222873f2d1497e1e0fb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:25:10.759161 21717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c1e1a2ed8c4222873f2d1497e1e0fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } } peers { permanent_uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } } }
I20260924 00:25:10.759268 21717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c1e1a2ed8c4222873f2d1497e1e0fb [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:25:10.759328 21559 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:25:10.759364 21533 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:25:10.759418 21533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:25:10.760425 21533 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } } peers { permanent_uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } }
I20260924 00:25:10.760730 21533 leader_election.cc:288] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58 [CANDIDATE]: Term 1 election: Requested vote from peers 3610dcbc59f94e56bf1b79a2a38b8e7e (127.0.98.61:43921), 55c1e1a2ed8c4222873f2d1497e1e0fb (127.0.98.60:35131)
I20260924 00:25:10.761220 21683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb"
I20260924 00:25:10.761345 21683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55c1e1a2ed8c4222873f2d1497e1e0fb [term 0 FOLLOWER]: Advancing to term 1
W20260924 00:25:10.761987 21735 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55c1e1a2ed8c4222873f2d1497e1e0fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:25:10.762041 21735 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 00:25:10.762061 21740 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:25:10.762120 21740 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:25:10.762795 21683 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 55c1e1a2ed8c4222873f2d1497e1e0fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce3aa9c0b0f34b628935a58bd1ca5e58 in term 1.
I20260924 00:25:10.763012 21510 leader_election.cc:302] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55c1e1a2ed8c4222873f2d1497e1e0fb, ce3aa9c0b0f34b628935a58bd1ca5e58; no voters:
I20260924 00:25:10.762018 21491 master_runner.cc:186] Error getting master registration for 127.0.98.61:43921: OK, instance_id { permanent_uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e" instance_seqno: 1790209510555058 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 00:25:10.764870 21491 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 00:25:10.764940 21718 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } } peers { permanent_uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } } }
I20260924 00:25:10.765028 21718 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:25:10.765070 21718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:25:10.765110 21718 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58 [term 1 LEADER]: Becoming Leader. State: Replica: ce3aa9c0b0f34b628935a58bd1ca5e58, State: Running, Role: LEADER
I20260924 00:25:10.765189 21718 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } } peers { permanent_uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } }
I20260924 00:25:10.765681 21743 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce3aa9c0b0f34b628935a58bd1ca5e58. Latest consensus state: current_term: 1 leader_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } } peers { permanent_uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } } }
I20260924 00:25:10.765890 21743 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58 [sys.catalog]: This master's current role is: LEADER
I20260924 00:25:10.766719 21749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:25:10.764954 21641 master_runner.cc:186] Error getting master registration for 127.0.98.61:43921: OK, instance_id { permanent_uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e" instance_seqno: 1790209510555058 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 00:25:10.767485 21641 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 00:25:10.767305 21749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:25:10.771234 21683 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 55c1e1a2ed8c4222873f2d1497e1e0fb [term 1 FOLLOWER]: Refusing update from remote peer ce3aa9c0b0f34b628935a58bd1ca5e58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:25:10.771787 21743 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:25:10.773566 21717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c1e1a2ed8c4222873f2d1497e1e0fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce3aa9c0b0f34b628935a58bd1ca5e58. Latest consensus state: current_term: 1 leader_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } } peers { permanent_uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } } }
I20260924 00:25:10.773705 21717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c1e1a2ed8c4222873f2d1497e1e0fb [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:25:10.775771 21717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c1e1a2ed8c4222873f2d1497e1e0fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } } peers { permanent_uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } } }
I20260924 00:25:10.775852 21717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c1e1a2ed8c4222873f2d1497e1e0fb [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:25:10.776067 21752 mvcc.cc:205] Tried to move back new op lower bound from 7332698156116586496 to 7332698156095070208. Current Snapshot: MvccSnapshot[applied={T|T < 7332698156116586496}]
I20260924 00:25:10.778867 21562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.766306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50354 (local address 127.0.98.62:37507)
0924 00:25:10.766346 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.766348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.767204 (+ 856us) server_negotiation.cc:407] Connection header received
0924 00:25:10.767649 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.767650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.767661 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.767694 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.768626 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.769471 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.770616 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.770971 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.771049 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.774902 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:10.774903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:10.774904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:10.774911 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:10.777139 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.777568 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.777570 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.777570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.777636 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.777842 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.777843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.777843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.777976 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:10.778086 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:10.778663 (+ 577us) server_negotiation.cc:299] Negotiation successful
0924 00:25:10.778753 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 00:25:10.779147 21751 mvcc.cc:205] Tried to move back new op lower bound from 7332698156116586496 to 7332698156095070208. Current Snapshot: MvccSnapshot[applied={T|T < 7332698156116586496}]
I20260924 00:25:10.779873 21718 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } } peers { permanent_uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } } }
I20260924 00:25:10.779979 21718 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58 [sys.catalog]: This master's current role is: LEADER
I20260924 00:25:10.781750 21749 catalog_manager.cc:1347] Generated new cluster ID: a58da586f3134745a3851d9b1217f062
I20260924 00:25:10.781801 21749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:25:10.790160 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.773278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44252 (local address 127.0.98.61:43921)
0924 00:25:10.773802 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.773804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.773813 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:25:10.773838 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.773839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.773849 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.773896 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.776547 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.777194 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.779272 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.779610 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.779657 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.781385 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:10.781387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:10.781387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:10.781395 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:10.783479 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.788820 (+ 5341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.788822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.788823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.788894 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.789492 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.789494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.789494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.789638 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:10.789735 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:10.789948 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 00:25:10.790039 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496}
I20260924 00:25:10.790645 21749 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:25:10.791429 21749 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:25:10.792930 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.773290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33652 (local address 127.0.98.60:35131)
0924 00:25:10.773328 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.773330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.773709 (+ 379us) server_negotiation.cc:407] Connection header received
0924 00:25:10.773800 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.773801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.773814 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.773846 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.776224 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.777027 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.780080 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.780449 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.780497 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.789714 (+ 9217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:10.789716 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:10.789716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:10.789723 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:10.791432 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.792141 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.792142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.792142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.792218 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.792394 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.792395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.792395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.792506 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:10.792569 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:10.792749 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 00:25:10.792823 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 00:25:10.797411 21749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58: Generated new TSK 0
I20260924 00:25:10.797591 21749 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 00:25:10.798429 21634 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } attempt: 3
Sep 24 00:25:10 dist-test-slave-777x krb5kdc[21478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 00:25:10.808816 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.798691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35131 (local address 127.0.0.1:33660)
0924 00:25:10.798791 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:10.798803 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:25:10.798843 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:25:10.800443 (+ 1600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:25:10.800444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:25:10.800628 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:25:10.800680 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:10.800682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.801559 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.801560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:10.802472 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:10.802474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.803484 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.803485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:10.803555 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.804033 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:25:10.804038 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:25:10.805629 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:25:10.807603 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:10.807604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:10.807605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:10.807826 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:10.808036 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:10.808036 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:10.808037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:10.808095 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:10.808436 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:25:10.808437 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:25:10.808501 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 00:25:10.808572 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 00:25:10.808683 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 00:25:10.809715 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:10.800270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33660 (local address 127.0.98.60:35131)
0924 00:25:10.800317 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.800319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.800327 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:25:10.800348 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.800348 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.800358 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.800418 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.800771 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.801479 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.803008 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:10.803350 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:10.803423 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:10.805731 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:10.805733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:10.805733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:10.805747 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:10.807504 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.807894 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.807895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.807896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.807959 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:10.808145 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:10.808146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:10.808146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:10.808318 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:10.808388 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:10.809424 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0924 00:25:10.809564 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:25:10.811837 21634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3610dcbc59f94e56bf1b79a2a38b8e7e: Bootstrap starting.
I20260924 00:25:10.812655 21634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3610dcbc59f94e56bf1b79a2a38b8e7e: Neither blocks nor log segments found. Creating new log.
I20260924 00:25:10.812987 21634 log.cc:824] T 00000000000000000000000000000000 P 3610dcbc59f94e56bf1b79a2a38b8e7e: Log is configured to *not* fsync() on all Append() calls
I20260924 00:25:10.821045 21634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3610dcbc59f94e56bf1b79a2a38b8e7e: No bootstrap required, opened a new log
I20260924 00:25:10.823693 21634 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3610dcbc59f94e56bf1b79a2a38b8e7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } } peers { permanent_uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } }
I20260924 00:25:10.824097 21634 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3610dcbc59f94e56bf1b79a2a38b8e7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3610dcbc59f94e56bf1b79a2a38b8e7e, State: Initialized, Role: FOLLOWER
I20260924 00:25:10.824345 21634 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3610dcbc59f94e56bf1b79a2a38b8e7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } } peers { permanent_uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } }
I20260924 00:25:10.824815 21634 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3610dcbc59f94e56bf1b79a2a38b8e7e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:25:10.824890 21607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3610dcbc59f94e56bf1b79a2a38b8e7e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:25:10.826004 21607 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3610dcbc59f94e56bf1b79a2a38b8e7e [term 1 FOLLOWER]: Refusing update from remote peer ce3aa9c0b0f34b628935a58bd1ca5e58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:25:10.826225 21608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e"
I20260924 00:25:10.827080 21608 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3610dcbc59f94e56bf1b79a2a38b8e7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce3aa9c0b0f34b628935a58bd1ca5e58 in term 1.
I20260924 00:25:10.827813 21743 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 00:25:10.828898 21639 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:10.827698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44260 (local address 127.0.98.61:43921)
0924 00:25:10.827751 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.827754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.828147 (+ 393us) server_negotiation.cc:407] Connection header received
0924 00:25:10.828327 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.828329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.828339 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.828373 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.828837 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44260: BlockingRecv error: recv got EOF from 127.0.0.1:44260 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:25:10.830675 21783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3610dcbc59f94e56bf1b79a2a38b8e7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce3aa9c0b0f34b628935a58bd1ca5e58. Latest consensus state: current_term: 1 leader_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } } peers { permanent_uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } } }
I20260924 00:25:10.830864 21783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3610dcbc59f94e56bf1b79a2a38b8e7e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:25:10.830962 21783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3610dcbc59f94e56bf1b79a2a38b8e7e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } } peers { permanent_uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } } }
I20260924 00:25:10.831063 21783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3610dcbc59f94e56bf1b79a2a38b8e7e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:25:10.831194 21763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3610dcbc59f94e56bf1b79a2a38b8e7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } } peers { permanent_uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } } }
I20260924 00:25:10.831283 21763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3610dcbc59f94e56bf1b79a2a38b8e7e [sys.catalog]: This master's current role is: FOLLOWER
W20260924 00:25:10.831794 21778 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3610dcbc59f94e56bf1b79a2a38b8e7e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:25:10.831847 21778 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:25:10.832448 21781 mvcc.cc:205] Tried to move back new op lower bound from 7332698156220309504 to 7332698156095070208. Current Snapshot: MvccSnapshot[applied={T|T < 7332698156116586496}]
W20260924 00:25:10.833349 21714 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:10.832386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33666 (local address 127.0.98.60:35131)
0924 00:25:10.832430 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.832432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.832717 (+ 285us) server_negotiation.cc:407] Connection header received
0924 00:25:10.832754 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.832755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.832766 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.832839 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.833286 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33666: BlockingRecv error: recv got EOF from 127.0.0.1:33666 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 00:25:10.837278 21562 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:10.836529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50370 (local address 127.0.98.62:37507)
0924 00:25:10.836571 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.836573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.836755 (+ 182us) server_negotiation.cc:407] Connection header received
0924 00:25:10.836779 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.836780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.836790 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.836833 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.837195 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50370: BlockingRecv error: recv got EOF from 127.0.0.1:50370 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 00:25:10.841166 21639 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:10.840500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44276 (local address 127.0.98.61:43921)
0924 00:25:10.840542 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.840544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.840632 (+ 88us) server_negotiation.cc:407] Connection header received
0924 00:25:10.840740 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.840741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.840760 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.840791 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.841116 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44276: BlockingRecv error: recv got EOF from 127.0.0.1:44276 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 00:25:10.843962 21639 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:10.843396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44286 (local address 127.0.98.61:43921)
0924 00:25:10.843469 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.843471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.843480 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:25:10.843514 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.843515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.843526 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.843602 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.843891 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44286: BlockingRecv error: recv got EOF from 127.0.0.1:44286 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 00:25:10.847612 21639 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:10.846791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44298 (local address 127.0.98.61:43921)
0924 00:25:10.846827 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.846829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.847105 (+ 276us) server_negotiation.cc:407] Connection header received
0924 00:25:10.847201 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.847202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.847212 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.847241 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.847565 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44298: BlockingRecv error: recv got EOF from 127.0.0.1:44298 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 00:25:10.853608 21639 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:10.852799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44300 (local address 127.0.98.61:43921)
0924 00:25:10.852837 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.852840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.853067 (+ 227us) server_negotiation.cc:407] Connection header received
0924 00:25:10.853163 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.853164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.853175 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.853206 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.853545 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44300: BlockingRecv error: recv got EOF from 127.0.0.1:44300 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 00:25:10.869524 21639 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:10.868518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44304 (local address 127.0.98.61:43921)
0924 00:25:10.868566 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.868568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.868859 (+ 291us) server_negotiation.cc:407] Connection header received
0924 00:25:10.868974 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.868975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.868988 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.869021 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.869458 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44304: BlockingRecv error: recv got EOF from 127.0.0.1:44304 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 00:25:10.887976 21639 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:10.887015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44318 (local address 127.0.98.61:43921)
0924 00:25:10.887061 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.887063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.887349 (+ 286us) server_negotiation.cc:407] Connection header received
0924 00:25:10.887476 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.887477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.887490 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.887522 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.887916 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44318: BlockingRecv error: recv got EOF from 127.0.0.1:44318 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 00:25:10.922534 21639 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:10.921579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44332 (local address 127.0.98.61:43921)
0924 00:25:10.921624 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.921626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.921911 (+ 285us) server_negotiation.cc:407] Connection header received
0924 00:25:10.922019 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.922020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.922032 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.922063 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.922470 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44332: BlockingRecv error: recv got EOF from 127.0.0.1:44332 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 00:25:10.988715 21639 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:10.987895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44342 (local address 127.0.98.61:43921)
0924 00:25:10.987946 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:10.987949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:10.988099 (+ 150us) server_negotiation.cc:407] Connection header received
0924 00:25:10.988184 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:10.988186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:10.988200 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:10.988287 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:10.988631 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44342: BlockingRecv error: recv got EOF from 127.0.0.1:44342 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 00:25:11.118680 21639 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:11.117941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44356 (local address 127.0.98.61:43921)
0924 00:25:11.118051 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:11.118053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:11.118067 (+ 14us) server_negotiation.cc:407] Connection header received
0924 00:25:11.118129 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:11.118131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:11.118142 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:11.118192 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:11.118598 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44356: BlockingRecv error: recv got EOF from 127.0.0.1:44356 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 00:25:11.247996 21718 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce3aa9c0b0f34b628935a58bd1ca5e58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37507 } } peers { permanent_uuid: "3610dcbc59f94e56bf1b79a2a38b8e7e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43921 } } peers { permanent_uuid: "55c1e1a2ed8c4222873f2d1497e1e0fb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35131 } } }
I20260924 00:25:11.248139 21718 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce3aa9c0b0f34b628935a58bd1ca5e58 [sys.catalog]: This master's current role is: LEADER
W20260924 00:25:11.376943 21639 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:11.376088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44364 (local address 127.0.98.61:43921)
0924 00:25:11.376285 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:11.376288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:11.376303 (+ 15us) server_negotiation.cc:407] Connection header received
0924 00:25:11.376338 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:11.376339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:11.376359 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:11.376427 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:11.376863 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44364: BlockingRecv error: recv got EOF from 127.0.0.1:44364 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 00:25:11.762486 21735 catalog_manager.cc:1259] Loaded cluster ID: a58da586f3134745a3851d9b1217f062
I20260924 00:25:11.762573 21735 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 55c1e1a2ed8c4222873f2d1497e1e0fb: loading cluster ID for follower catalog manager: success
I20260924 00:25:11.763556 21735 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 55c1e1a2ed8c4222873f2d1497e1e0fb: acquiring CA information for follower catalog manager: success
I20260924 00:25:11.763926 21735 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 55c1e1a2ed8c4222873f2d1497e1e0fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 00:25:11.832401 21778 catalog_manager.cc:1259] Loaded cluster ID: a58da586f3134745a3851d9b1217f062
I20260924 00:25:11.832486 21778 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3610dcbc59f94e56bf1b79a2a38b8e7e: loading cluster ID for follower catalog manager: success
I20260924 00:25:11.833518 21778 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3610dcbc59f94e56bf1b79a2a38b8e7e: acquiring CA information for follower catalog manager: success
I20260924 00:25:11.833842 21778 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3610dcbc59f94e56bf1b79a2a38b8e7e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 00:25:11.893497 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:11.890306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44372 (local address 127.0.98.61:43921)
0924 00:25:11.890448 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:11.890451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:11.890462 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:25:11.890547 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:11.890548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:11.890628 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:11.890725 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:25:11.890900 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:11.891469 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:11.892430 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:11.892678 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:11.892768 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:11.892995 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:25:11.893132 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:25:11.893287 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:11.893307 (+ 20us) server_negotiation.cc:299] Negotiation successful
0924 00:25:11.893332 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
I20260924 00:25:11.898870 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:11.895954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33676 (local address 127.0.98.60:35131)
0924 00:25:11.896093 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:11.896095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:11.896104 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:25:11.896222 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:11.896223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:11.896280 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:11.896439 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:25:11.896578 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:11.897143 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:11.897966 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:11.898193 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:11.898292 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:11.898482 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:25:11.898602 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:25:11.898703 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:11.898738 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 00:25:11.898762 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260924 00:25:11.903964 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:11.900882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44384 (local address 127.0.98.61:43921)
0924 00:25:11.900976 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:11.900979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:11.901300 (+ 321us) server_negotiation.cc:407] Connection header received
0924 00:25:11.901391 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:11.901392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:11.901407 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:11.901464 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:25:11.901718 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:11.902256 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:11.903181 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:11.903400 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:11.903488 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:11.903665 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:25:11.903700 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:25:11.903726 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:11.903834 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 00:25:11.903858 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 00:25:11.904552 21850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:11.900882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.0.98.60:35131)
0924 00:25:11.900931 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:11.900934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:11.901176 (+ 242us) server_negotiation.cc:407] Connection header received
0924 00:25:11.901217 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:11.901218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:11.901228 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:11.901302 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:25:11.901469 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:11.902225 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:11.903643 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:11.903868 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:11.903984 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:11.904221 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:25:11.904253 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:25:11.904284 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:11.904436 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 00:25:11.904462 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 00:25:11.904790 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:11.900893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50374 (local address 127.0.98.62:37507)
0924 00:25:11.901038 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:11.901040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:11.901113 (+ 73us) server_negotiation.cc:407] Connection header received
0924 00:25:11.901167 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:11.901167 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:11.901223 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:11.901400 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:25:11.901624 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:11.902365 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:11.903607 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:11.903924 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:11.903982 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:11.904273 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:25:11.904393 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:25:11.904455 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:11.904618 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 00:25:11.904704 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260924 00:25:11.906903 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 21491
I20260924 00:25:11.913571 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 21566
I20260924 00:25:11.919051 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 21641
2026-09-24T00:25:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2814 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 00:25:11 dist-test-slave-777x krb5kdc[21861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 00:25:11 dist-test-slave-777x krb5kdc[21861](info): set up 2 sockets
Sep 24 00:25:11 dist-test-slave-777x krb5kdc[21861](info): commencing operation
krb5kdc: starting...
W20260924 00:25:13.032523 392 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/test-user.keytab.
Sep 24 00:25:13 dist-test-slave-777x krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T00:25:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T00:25:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 00:25:13.163852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44387
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44387,127.0.98.61:35883,127.0.98.60:44325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 00:25:13.245229 21874 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 00:25:13.245484 21874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:25:13.245514 21874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:25:13.245565 21874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:25:13.246701 21874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:25:13.246767 21874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:25:13.246791 21874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:25:13.246812 21874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:25:13.246874 21874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:25:13.248451 21874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44387,127.0.98.61:35883,127.0.98.60:44325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:25:13.248723 21874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:25:13.248978 21874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 00:25:13.254019 21874 server_base.cc:1034] running on GCE node
W20260924 00:25:13.253976 21879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:25:13.253998 21882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:25:13.253998 21880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:25:13.254454 21874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:25:13.254762 21874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:25:13.255936 21874 hybrid_clock.cc:648] HybridClock initialized: now 1790209513255926 us; error 34 us; skew 500 ppm
Sep 24 00:25:13 dist-test-slave-777x krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:25:13.258491 21874 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 00:25:13.258965 21874 webserver.cc:460] Webserver started at http://127.0.98.62:44131/ using document root <none> and password file <none>
I20260924 00:25:13.259202 21874 fs_manager.cc:359] Metadata directory not provided
I20260924 00:25:13.259267 21874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:25:13.259433 21874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:25:13.260617 21874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-0/data/instance:
uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e"
format_stamp: "Formatted at 2026-09-24 00:25:13 on dist-test-slave-777x"
server_key: "d0101a1d87553738848e53f51eceb2a9"
server_key_iv: "9d5886c5b50629bed545370a09736d7d"
server_key_version: "encryptionkey@0"
I20260924 00:25:13.260978 21874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-0/wal/instance:
uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e"
format_stamp: "Formatted at 2026-09-24 00:25:13 on dist-test-slave-777x"
server_key: "d0101a1d87553738848e53f51eceb2a9"
server_key_iv: "9d5886c5b50629bed545370a09736d7d"
server_key_version: "encryptionkey@0"
I20260924 00:25:13.262349 21874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:25:13.263245 21889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:25:13.263449 21874 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:25:13.263549 21874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-0/wal
uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e"
format_stamp: "Formatted at 2026-09-24 00:25:13 on dist-test-slave-777x"
server_key: "d0101a1d87553738848e53f51eceb2a9"
server_key_iv: "9d5886c5b50629bed545370a09736d7d"
server_key_version: "encryptionkey@0"
I20260924 00:25:13.263634 21874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:25:13.277613 21874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:25:13.279842 21874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:25:13.280025 21874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:25:13.284281 21874 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44387
I20260924 00:25:13.284421 21941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44387 every 8 connection(s)
I20260924 00:25:13.284737 21874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-0/data/info.pb
I20260924 00:25:13.285825 21942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:25:13.287541 21942 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:25:13.288925 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 21874
I20260924 00:25:13.289005 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-0/wal/instance
I20260924 00:25:13.289288 392 external_mini_cluster.cc:1428] Setting key fa3a3037ad7f1d12aea479df34e49883
Sep 24 00:25:13 dist-test-slave-777x krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260924 00:25:13.299299 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.288322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40942 (local address 127.0.98.62:44387)
0924 00:25:13.288501 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.288506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.288579 (+ 73us) server_negotiation.cc:407] Connection header received
0924 00:25:13.289160 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.289167 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.289185 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.289286 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.289896 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.291091 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.292282 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.292855 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.292983 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.295602 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:13.295611 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:13.295612 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:13.295620 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:13.297633 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.298058 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.298061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.298061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.298179 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.298522 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.298523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.298523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.298751 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:13.298832 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:13.299029 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 00:25:13.299127 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":115,"threads_started":1}
I20260924 00:25:13.299441 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.288284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.0.1:40942)
0924 00:25:13.288447 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.288494 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 00:25:13.288955 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:25:13.289294 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:25:13.289297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:25:13.289629 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:25:13.289790 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:13.289795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.291308 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.291310 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:13.292100 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:13.292102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.292988 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.292990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:13.293065 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.293693 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:25:13.293703 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:25:13.295497 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:25:13.297738 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:13.297746 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:13.297748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:13.297975 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:13.298293 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:13.298294 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:13.298295 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:13.298426 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:13.298828 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:25:13.298835 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:25:13.298906 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 00:25:13.298993 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 00:25:13.299251 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":75,"thread_start_us":86,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260924 00:25:13.300446 21942 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:25:13.300855 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.300695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35883 (local address 127.0.0.1:41984)
0924 00:25:13.300765 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.300821 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 00:25:13.301003 21891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35883: connect: Connection refused (error 111)
I20260924 00:25:13.301507 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.301397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35883 (local address 127.0.0.1:41994)
0924 00:25:13.301447 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.301476 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 00:25:13.301689 21942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35883: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35883: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 00:25:13.316486 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35883
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44387,127.0.98.61:35883,127.0.98.60:44325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:25:13.350978 21942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } attempt: 1
I20260924 00:25:13.351522 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.351365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35883 (local address 127.0.0.1:42000)
0924 00:25:13.351442 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.351489 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 00:25:13.351902 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.351799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35883 (local address 127.0.0.1:42006)
0924 00:25:13.351849 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.351875 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 00:25:13.351990 21942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35883: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35883: connect: Connection refused (error 111)
W20260924 00:25:13.400976 21949 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 00:25:13.401319 21949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:25:13.401356 21949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:25:13.401398 21949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:25:13.402597 21949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:25:13.402671 21949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:25:13.402694 21949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:25:13.402716 21949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:25:13.402746 21949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:25:13.404399 21949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44387,127.0.98.61:35883,127.0.98.60:44325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:25:13.404703 21949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:25:13.405010 21949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:25:13.410136 21957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:25:13.410125 21955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:25:13.410290 21949 server_base.cc:1034] running on GCE node
W20260924 00:25:13.410156 21954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:25:13.410703 21949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:25:13.411087 21949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:25:13.412261 21949 hybrid_clock.cc:648] HybridClock initialized: now 1790209513412239 us; error 46 us; skew 500 ppm
Sep 24 00:25:13 dist-test-slave-777x krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:25:13.415032 21949 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 00:25:13.415647 21949 webserver.cc:460] Webserver started at http://127.0.98.61:40961/ using document root <none> and password file <none>
I20260924 00:25:13.415907 21949 fs_manager.cc:359] Metadata directory not provided
I20260924 00:25:13.415983 21949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:25:13.416147 21949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:25:13.417397 21949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-1/data/instance:
uuid: "455b792d08a74a29bfd757771f3f29a2"
format_stamp: "Formatted at 2026-09-24 00:25:13 on dist-test-slave-777x"
server_key: "2d26c58d73e77defc8ada3324a255046"
server_key_iv: "1ec7553c645f1cb084a69f5303aad1ce"
server_key_version: "encryptionkey@0"
I20260924 00:25:13.417845 21949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-1/wal/instance:
uuid: "455b792d08a74a29bfd757771f3f29a2"
format_stamp: "Formatted at 2026-09-24 00:25:13 on dist-test-slave-777x"
server_key: "2d26c58d73e77defc8ada3324a255046"
server_key_iv: "1ec7553c645f1cb084a69f5303aad1ce"
server_key_version: "encryptionkey@0"
I20260924 00:25:13.419564 21949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 00:25:13.420130 21942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } attempt: 2
I20260924 00:25:13.420687 21964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:25:13.420715 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.420500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35883 (local address 127.0.0.1:42012)
0924 00:25:13.420619 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.420670 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 00:25:13.420938 21949 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:25:13.421065 21949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-1/wal
uuid: "455b792d08a74a29bfd757771f3f29a2"
format_stamp: "Formatted at 2026-09-24 00:25:13 on dist-test-slave-777x"
server_key: "2d26c58d73e77defc8ada3324a255046"
server_key_iv: "1ec7553c645f1cb084a69f5303aad1ce"
server_key_version: "encryptionkey@0"
I20260924 00:25:13.421187 21949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:25:13.421218 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.421091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35883 (local address 127.0.0.1:42020)
0924 00:25:13.421149 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.421188 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 00:25:13.421402 21942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:35883: Network error: Client connection negotiation failed: client connection to 127.0.98.61:35883: connect: Connection refused (error 111)
I20260924 00:25:13.439810 21949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:25:13.442608 21949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:25:13.442737 21949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:25:13.447432 21949 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35883
I20260924 00:25:13.447471 22016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35883 every 8 connection(s)
I20260924 00:25:13.447883 21949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-1/data/info.pb
I20260924 00:25:13.449077 22017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:25:13.450971 22017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:25:13.452011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 21949
I20260924 00:25:13.452142 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-1/wal/instance
I20260924 00:25:13.452447 392 external_mini_cluster.cc:1428] Setting key 070cefa759cd57c5e2878918600f7a6c
Sep 24 00:25:13 dist-test-slave-777x krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260924 00:25:13.461190 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.451791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40950 (local address 127.0.98.62:44387)
0924 00:25:13.452082 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.452085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.452230 (+ 145us) server_negotiation.cc:407] Connection header received
0924 00:25:13.452819 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.452820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.452835 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.452885 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.453521 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.454023 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.455186 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.455553 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.455604 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.458051 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:13.458053 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:13.458054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:13.458061 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:13.459591 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.460015 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.460016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.460017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.460083 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.460353 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.460354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.460355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.460521 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:13.460608 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:13.461050 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 00:25:13.461125 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259}
I20260924 00:25:13.461321 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.451738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.0.1:40950)
0924 00:25:13.452019 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.452058 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 00:25:13.452706 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:25:13.452959 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:25:13.452963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:25:13.453223 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:25:13.453426 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:13.453430 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.454121 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.454122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:13.455072 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:13.455076 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.455720 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.455721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:13.455902 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.456528 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:25:13.456537 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:25:13.457948 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:25:13.459686 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:13.459693 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:13.459696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:13.459934 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:13.460150 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:13.460151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:13.460152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:13.460281 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:13.460621 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:25:13.460630 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:25:13.460839 (+ 209us) client_negotiation.cc:769] Sending connection context
0924 00:25:13.460960 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 00:25:13.461104 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":155,"threads_started":1}
I20260924 00:25:13.461927 22017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 00:25:13 dist-test-slave-777x krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:25:13.472018 22021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.462300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42036 (local address 127.0.98.61:35883)
0924 00:25:13.462476 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.462480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.462530 (+ 50us) server_negotiation.cc:407] Connection header received
0924 00:25:13.462633 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.462658 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.462669 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.462732 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.463091 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.464260 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.465376 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.465909 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.465991 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.468517 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:13.468525 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:13.468525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:13.468532 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:13.470594 (+ 2062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.471077 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.471081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.471081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.471143 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.471394 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.471395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.471395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.471573 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:13.471659 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:13.471852 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 00:25:13.471936 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260924 00:25:13.472301 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.462207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35883 (local address 127.0.0.1:42036)
0924 00:25:13.462444 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.462455 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:25:13.462541 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:25:13.462784 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:25:13.462785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:25:13.462917 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:25:13.462979 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:13.462981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.464337 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.464339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:13.465200 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:13.465204 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.466007 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.466008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:13.466075 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.466670 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:25:13.466675 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:25:13.468423 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:25:13.470701 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:13.470702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:13.470703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:13.470975 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:13.471218 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:13.471219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:13.471219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:13.471293 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:13.471659 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:25:13.471661 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:25:13.471721 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 00:25:13.471830 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 00:25:13.472039 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":80}
I20260924 00:25:13.473146 22017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:25:13.473577 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.473471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44325 (local address 127.0.0.1:58894)
0924 00:25:13.473507 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.473557 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260924 00:25:13.473726 21966 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44325: connect: Connection refused (error 111)
I20260924 00:25:13.474251 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.474126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44325 (local address 127.0.0.1:58910)
0924 00:25:13.474189 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.474220 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 00:25:13.474449 22017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44325: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44325: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 00:25:13.484290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYEWBIr/build/release/bin/kudu
/tmp/dist-test-taskYEWBIr/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44325
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44387,127.0.98.61:35883,127.0.98.60:44325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 00:25:13.512562 21942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } attempt: 3
Sep 24 00:25:13 dist-test-slave-777x krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:25:13.520831 22021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.513133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42038 (local address 127.0.98.61:35883)
0924 00:25:13.513207 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.513210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.513223 (+ 13us) server_negotiation.cc:407] Connection header received
0924 00:25:13.513332 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.513333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.513344 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.513433 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.513805 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.514379 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.515171 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.515443 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.515544 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.517830 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:13.517832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:13.517832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:13.517842 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:13.519470 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.519902 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.519903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.519904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.519968 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.520218 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.520219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.520219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.520355 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:13.520444 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:13.520627 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 00:25:13.520730 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 00:25:13.520877 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.513049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35883 (local address 127.0.0.1:42038)
0924 00:25:13.513121 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.513132 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:25:13.513198 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:25:13.513439 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:25:13.513440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:25:13.513628 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:25:13.513696 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:13.513698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.514471 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.514473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:13.515071 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:13.515073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.515544 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.515545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:13.515647 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.516294 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:25:13.516301 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:25:13.517711 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:25:13.519579 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:13.519580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:13.519581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:13.519797 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:13.520050 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:13.520051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:13.520051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:13.520117 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:13.520448 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:25:13.520449 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:25:13.520504 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 00:25:13.520598 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 00:25:13.520721 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 00:25:13.521373 21942 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:25:13.521744 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.521649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44325 (local address 127.0.0.1:58914)
0924 00:25:13.521687 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.521723 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 00:25:13.522060 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.521967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44325 (local address 127.0.0.1:58918)
0924 00:25:13.522015 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.522039 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 00:25:13.522156 21942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44325: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44325: connect: Connection refused (error 111)
I20260924 00:25:13.523748 22017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } attempt: 1
I20260924 00:25:13.524310 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.524135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44325 (local address 127.0.0.1:58934)
0924 00:25:13.524243 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.524274 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 00:25:13.524739 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.524588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44325 (local address 127.0.0.1:58946)
0924 00:25:13.524646 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.524701 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 00:25:13.524873 22017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44325: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44325: connect: Connection refused (error 111)
I20260924 00:25:13.553350 21942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } attempt: 1
I20260924 00:25:13.553913 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.553752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44325 (local address 127.0.0.1:58948)
0924 00:25:13.553831 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.553874 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 00:25:13.554281 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.554182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44325 (local address 127.0.0.1:58962)
0924 00:25:13.554230 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.554259 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 00:25:13.554371 21942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44325: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44325: connect: Connection refused (error 111)
W20260924 00:25:13.571722 22024 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 00:25:13.572037 22024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 00:25:13.572080 22024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 00:25:13.572122 22024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 00:25:13.573369 22024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 00:25:13.573441 22024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 00:25:13.573465 22024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 00:25:13.573487 22024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 00:25:13.573518 22024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 00:25:13.575168 22024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44387,127.0.98.61:35883,127.0.98.60:44325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 23:03:17 UTC on ccycloud.quasar-qwsgns.root.comops.site
build id 4668911
I20260924 00:25:13.575477 22024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 00:25:13.575721 22024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 00:25:13.581231 22032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:25:13.581254 22029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 00:25:13.581311 22030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 00:25:13.581504 22024 server_base.cc:1034] running on GCE node
I20260924 00:25:13.581738 22024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 00:25:13.582010 22024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 00:25:13.583178 22024 hybrid_clock.cc:648] HybridClock initialized: now 1790209513583162 us; error 33 us; skew 500 ppm
Sep 24 00:25:13 dist-test-slave-777x krb5kdc[21861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 00:25:13.586444 22024 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 00:25:13.587041 22024 webserver.cc:460] Webserver started at http://127.0.98.60:37561/ using document root <none> and password file <none>
I20260924 00:25:13.587333 22024 fs_manager.cc:359] Metadata directory not provided
I20260924 00:25:13.587397 22024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 00:25:13.587556 22024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 00:25:13.588864 22024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-2/data/instance:
uuid: "64394c7a27e34d098530337bd2e35d20"
format_stamp: "Formatted at 2026-09-24 00:25:13 on dist-test-slave-777x"
server_key: "a5351ff7af4d5cc7e7ade6c40cdae7ee"
server_key_iv: "f42c9de532253896eb82f3247c573e7c"
server_key_version: "encryptionkey@0"
I20260924 00:25:13.589267 22024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-2/wal/instance:
uuid: "64394c7a27e34d098530337bd2e35d20"
format_stamp: "Formatted at 2026-09-24 00:25:13 on dist-test-slave-777x"
server_key: "a5351ff7af4d5cc7e7ade6c40cdae7ee"
server_key_iv: "f42c9de532253896eb82f3247c573e7c"
server_key_version: "encryptionkey@0"
I20260924 00:25:13.590838 22024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 00:25:13.591672 22039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 00:25:13.591871 22024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 00:25:13.591976 22024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-2/wal
uuid: "64394c7a27e34d098530337bd2e35d20"
format_stamp: "Formatted at 2026-09-24 00:25:13 on dist-test-slave-777x"
server_key: "a5351ff7af4d5cc7e7ade6c40cdae7ee"
server_key_iv: "f42c9de532253896eb82f3247c573e7c"
server_key_version: "encryptionkey@0"
I20260924 00:25:13.592067 22024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 00:25:13.593043 22017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } attempt: 2
I20260924 00:25:13.593531 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.593366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44325 (local address 127.0.0.1:58978)
0924 00:25:13.593434 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.593479 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 00:25:13.593915 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.593806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44325 (local address 127.0.0.1:58980)
0924 00:25:13.593856 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.593883 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 00:25:13.594074 22017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44325: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44325: connect: Connection refused (error 111)
I20260924 00:25:13.626472 22024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 00:25:13.629130 22024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 00:25:13.629253 22024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 00:25:13.629595 21942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } attempt: 2
I20260924 00:25:13.630121 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.629964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44325 (local address 127.0.0.1:58990)
0924 00:25:13.630045 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.630084 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 00:25:13.630587 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.630477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44325 (local address 127.0.0.1:59004)
0924 00:25:13.630530 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.630565 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 00:25:13.630677 21942 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44325: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44325: connect: Connection refused (error 111)
I20260924 00:25:13.633642 22024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44325
I20260924 00:25:13.633716 22091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44325 every 8 connection(s)
I20260924 00:25:13.634032 22024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-2/data/info.pb
I20260924 00:25:13.635046 22092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 00:25:13.636765 22092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:25:13.640323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYEWBIr/build/release/bin/kudu as pid 22024
I20260924 00:25:13.640452 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYEWBIr/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790209154318147-392-0/minicluster-data/master-2/wal/instance
I20260924 00:25:13.640663 392 external_mini_cluster.cc:1428] Setting key 8f1f35dd856776edcd87ccee26f0cdc4
Sep 24 00:25:13 dist-test-slave-777x krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 00:25:13 dist-test-slave-777x krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 00:25:13.647106 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.637510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40966 (local address 127.0.98.62:44387)
0924 00:25:13.637569 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.637573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.637764 (+ 191us) server_negotiation.cc:407] Connection header received
0924 00:25:13.638270 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.638272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.638283 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.638346 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.639046 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.639611 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.640845 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.641038 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.641130 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.643663 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:13.643665 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:13.643665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:13.643673 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:13.645469 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.645944 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.645946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.645946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.646032 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.646384 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.646385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.646385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.646499 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:13.646555 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:13.646950 (+ 395us) server_negotiation.cc:299] Negotiation successful
0924 00:25:13.647033 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 00:25:13.647459 22094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.637464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44387 (local address 127.0.0.1:40966)
0924 00:25:13.637635 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.637677 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 00:25:13.638128 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:25:13.638474 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:25:13.638478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:25:13.638723 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:25:13.638934 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:13.638938 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.639698 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.639699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:13.640753 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:13.640756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.641167 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.641169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:13.641362 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.641927 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:25:13.641940 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:25:13.643575 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:25:13.645566 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:13.645572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:13.645574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:13.645862 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:13.646111 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:13.646112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:13.646112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:13.646290 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:13.646578 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:25:13.646585 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:25:13.646770 (+ 185us) client_negotiation.cc:769] Sending connection context
0924 00:25:13.646904 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 00:25:13.647083 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
I20260924 00:25:13.648201 22092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } has no permanent_uuid. Determining permanent_uuid...
I20260924 00:25:13.650876 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.641103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59020 (local address 127.0.98.60:44325)
0924 00:25:13.641267 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.641272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.641320 (+ 48us) server_negotiation.cc:407] Connection header received
0924 00:25:13.641390 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.641391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.641407 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.641466 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.642371 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.643032 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.643995 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.644496 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.644604 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.647449 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:13.647456 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:13.647457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:13.647464 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:13.649558 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.649978 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.649981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.649981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.650033 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.650252 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.650252 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.650253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.650421 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:13.650582 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:13.650668 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 00:25:13.650727 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":97,"threads_started":1}
Sep 24 00:25:13 dist-test-slave-777x krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:25:13.656293 22021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.648615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42052 (local address 127.0.98.61:35883)
0924 00:25:13.648665 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.648667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.648675 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:25:13.648700 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.648701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.648711 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.648767 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.649063 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.649790 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.650793 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.651086 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.651164 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.653233 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:13.653235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:13.653235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:13.653242 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:13.654769 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.655202 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.655203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.655204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.655273 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.655600 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.655602 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.655602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.655731 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:13.655835 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:13.656061 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 00:25:13.656138 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:25:13.656278 22094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.648517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35883 (local address 127.0.0.1:42052)
0924 00:25:13.648576 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.648587 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 00:25:13.648625 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:25:13.648775 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:25:13.648776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:25:13.648942 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:25:13.648979 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:13.648981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.649870 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.649871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:13.650712 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:13.650715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.651165 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.651166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:13.651218 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.651704 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:25:13.651709 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:25:13.653149 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:25:13.654850 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:13.654851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:13.654852 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:13.655112 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:13.655427 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:13.655428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:13.655428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:13.655500 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:13.655841 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:25:13.655842 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:25:13.655901 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 00:25:13.655996 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 00:25:13.656111 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 00:25:13.656690 22092 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 00:25:13 dist-test-slave-777x krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 00:25:13.661123 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.652052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40980 (local address 127.0.98.62:44387)
0924 00:25:13.652105 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.652107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.652117 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:25:13.652144 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.652144 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.652152 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.652242 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.652765 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.653354 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.654394 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.654705 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.654784 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.657770 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:13.657772 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:13.657772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:13.657780 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:13.659564 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.660064 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.660066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.660066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.660136 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.660504 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.660505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.660505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.660634 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:13.660880 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:13.660923 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 00:25:13.661007 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 00:25:13 dist-test-slave-777x krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 00:25:13.665091 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.657084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59024 (local address 127.0.98.60:44325)
0924 00:25:13.657137 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.657139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.657147 (+ 8us) server_negotiation.cc:407] Connection header received
0924 00:25:13.657167 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.657168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.657186 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.657239 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.657549 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.658293 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.659191 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.659570 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.659647 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.661824 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:13.661826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:13.661827 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:13.661834 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:13.663563 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.664137 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.664138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.664138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.664233 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.664454 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.664462 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.664463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.664610 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:13.664686 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:13.664844 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 00:25:13.665005 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":35}
I20260924 00:25:13.665091 22094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.657008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44325 (local address 127.0.0.1:59024)
0924 00:25:13.657056 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.657066 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 00:25:13.657104 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:25:13.657266 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:25:13.657267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:25:13.657421 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:25:13.657466 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:13.657468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.658395 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.658396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:13.659100 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:13.659102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.659677 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.659678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:13.659759 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.660254 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:25:13.660259 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:25:13.661716 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:25:13.663665 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:13.663666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:13.663666 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:13.664005 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:13.664315 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:13.664316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:13.664316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:13.664380 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:13.664684 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:25:13.664685 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:25:13.664723 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 00:25:13.664779 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 00:25:13.664955 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 00:25:13.666877 22092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64394c7a27e34d098530337bd2e35d20: Bootstrap starting.
Sep 24 00:25:13 dist-test-slave-777x krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 00:25:13.667876 22092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64394c7a27e34d098530337bd2e35d20: Neither blocks nor log segments found. Creating new log.
I20260924 00:25:13.668249 22092 log.cc:824] T 00000000000000000000000000000000 P 64394c7a27e34d098530337bd2e35d20: Log is configured to *not* fsync() on all Append() calls
I20260924 00:25:13.669342 22092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64394c7a27e34d098530337bd2e35d20: No bootstrap required, opened a new log
I20260924 00:25:13.670931 22092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64394c7a27e34d098530337bd2e35d20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } peers { permanent_uuid: "455b792d08a74a29bfd757771f3f29a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } } peers { permanent_uuid: "64394c7a27e34d098530337bd2e35d20" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } }
I20260924 00:25:13.671067 22092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64394c7a27e34d098530337bd2e35d20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64394c7a27e34d098530337bd2e35d20, State: Initialized, Role: FOLLOWER
I20260924 00:25:13.671104 22021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.662055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42058 (local address 127.0.98.61:35883)
0924 00:25:13.662098 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.662100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.662109 (+ 9us) server_negotiation.cc:407] Connection header received
0924 00:25:13.662131 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.662134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.662149 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.662216 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.662802 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.663660 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.664853 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.665212 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.665302 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.667946 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:13.667947 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:13.667948 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:13.667957 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:13.669819 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.670257 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.670258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.670258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.670318 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.670542 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.670543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.670543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.670672 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:13.670849 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:13.670904 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 00:25:13.670990 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 00:25:13.671129 22092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64394c7a27e34d098530337bd2e35d20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } peers { permanent_uuid: "455b792d08a74a29bfd757771f3f29a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } } peers { permanent_uuid: "64394c7a27e34d098530337bd2e35d20" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } }
I20260924 00:25:13.671511 22092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64394c7a27e34d098530337bd2e35d20 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:25:13.671473 22098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64394c7a27e34d098530337bd2e35d20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } peers { permanent_uuid: "455b792d08a74a29bfd757771f3f29a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } } peers { permanent_uuid: "64394c7a27e34d098530337bd2e35d20" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } } }
I20260924 00:25:13.671576 22098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64394c7a27e34d098530337bd2e35d20 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:25:13.672940 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 00:25:13.673907 21916 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40980
W20260924 00:25:13.676154 22109 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64394c7a27e34d098530337bd2e35d20: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:25:13.676252 22109 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:25:13.674005 22024 master_runner.cc:186] Error getting master registration for 127.0.98.62:44387: OK, instance_id { permanent_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" instance_seqno: 1790209513283021 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 00:25:13.676774 22024 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 00:25:13.685231 22017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } attempt: 3
Sep 24 00:25:13 dist-test-slave-777x krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 00:25:13.693526 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.685711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44325 (local address 127.0.0.1:59026)
0924 00:25:13.685796 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.685810 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 00:25:13.685893 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:25:13.686058 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:25:13.686059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:25:13.686240 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:25:13.686287 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:13.686289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.687008 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.687009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:13.687594 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:13.687596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.688005 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.688006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:13.688079 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.688641 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:25:13.688647 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:25:13.690186 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:25:13.692291 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:13.692292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:13.692293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:13.692494 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:13.692740 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:13.692740 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:13.692741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:13.692811 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:13.693134 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:25:13.693135 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:25:13.693190 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 00:25:13.693287 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 00:25:13.693412 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 00:25:13.693569 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.685802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59026 (local address 127.0.98.60:44325)
0924 00:25:13.685870 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.685873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.685886 (+ 13us) server_negotiation.cc:407] Connection header received
0924 00:25:13.685970 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.685971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.685985 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.686041 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.686373 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.686907 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.687690 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.687911 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.688005 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.690289 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:13.690291 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:13.690291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:13.690300 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:13.692206 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.692579 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.692580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.692581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.692644 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.692892 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.692893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.692893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.693058 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:13.693144 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:13.693393 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 00:25:13.693475 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 00:25:13.695842 22017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 455b792d08a74a29bfd757771f3f29a2: Bootstrap starting.
I20260924 00:25:13.696658 22017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 455b792d08a74a29bfd757771f3f29a2: Neither blocks nor log segments found. Creating new log.
I20260924 00:25:13.696995 22017 log.cc:824] T 00000000000000000000000000000000 P 455b792d08a74a29bfd757771f3f29a2: Log is configured to *not* fsync() on all Append() calls
I20260924 00:25:13.697865 22017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 455b792d08a74a29bfd757771f3f29a2: No bootstrap required, opened a new log
I20260924 00:25:13.700048 22017 raft_consensus.cc:348] T 00000000000000000000000000000000 P 455b792d08a74a29bfd757771f3f29a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } peers { permanent_uuid: "455b792d08a74a29bfd757771f3f29a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } } peers { permanent_uuid: "64394c7a27e34d098530337bd2e35d20" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } }
I20260924 00:25:13.700253 22017 raft_consensus.cc:729] T 00000000000000000000000000000000 P 455b792d08a74a29bfd757771f3f29a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 455b792d08a74a29bfd757771f3f29a2, State: Initialized, Role: FOLLOWER
I20260924 00:25:13.700371 22017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 455b792d08a74a29bfd757771f3f29a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } peers { permanent_uuid: "455b792d08a74a29bfd757771f3f29a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } } peers { permanent_uuid: "64394c7a27e34d098530337bd2e35d20" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } }
I20260924 00:25:13.700726 22017 sys_catalog.cc:564] T 00000000000000000000000000000000 P 455b792d08a74a29bfd757771f3f29a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:25:13.700712 22112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 455b792d08a74a29bfd757771f3f29a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } peers { permanent_uuid: "455b792d08a74a29bfd757771f3f29a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } } peers { permanent_uuid: "64394c7a27e34d098530337bd2e35d20" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } } }
I20260924 00:25:13.700968 22112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 455b792d08a74a29bfd757771f3f29a2 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 00:25:13.703255 22123 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 455b792d08a74a29bfd757771f3f29a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:25:13.703325 22123 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:25:13.703675 21949 master_runner.cc:186] Error getting master registration for 127.0.98.62:44387: OK, instance_id { permanent_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" instance_seqno: 1790209513283021 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 00:25:13.707052 21949 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 00:25:13.729817 21942 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } attempt: 3
Sep 24 00:25:13 dist-test-slave-777x krb5kdc[21861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790209513, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 00:25:13.739557 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.730254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44325 (local address 127.0.0.1:59028)
0924 00:25:13.730320 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 00:25:13.730332 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 00:25:13.730381 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 00:25:13.730611 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 00:25:13.730612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 00:25:13.730837 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 00:25:13.730896 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:13.730898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.731941 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.731942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:13.732908 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 00:25:13.732911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.733536 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.733537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 00:25:13.733612 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.734223 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 00:25:13.734230 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 00:25:13.736029 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 00:25:13.738209 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:13.738211 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:13.738211 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:13.738488 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:13.738774 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 00:25:13.738775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 00:25:13.738776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 00:25:13.738867 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 00:25:13.739205 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 00:25:13.739206 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 00:25:13.739257 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 00:25:13.739327 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 00:25:13.739437 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 00:25:13.739992 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.730376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59028 (local address 127.0.98.60:44325)
0924 00:25:13.730434 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.730436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.730449 (+ 13us) server_negotiation.cc:407] Connection header received
0924 00:25:13.730484 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.730485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.730500 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.730594 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.730998 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.731832 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.733017 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.733362 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.733464 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.736138 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:13.736139 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:13.736140 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:13.736149 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:13.738097 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.738606 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.738608 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.738608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.738694 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.738953 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.738954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.738954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.739110 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:13.739176 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:13.739793 (+ 617us) server_negotiation.cc:299] Negotiation successful
0924 00:25:13.739889 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 00:25:13.742033 21942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e: Bootstrap starting.
I20260924 00:25:13.742868 21942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e: Neither blocks nor log segments found. Creating new log.
I20260924 00:25:13.743191 21942 log.cc:824] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e: Log is configured to *not* fsync() on all Append() calls
I20260924 00:25:13.744295 21942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e: No bootstrap required, opened a new log
I20260924 00:25:13.745890 21942 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } peers { permanent_uuid: "455b792d08a74a29bfd757771f3f29a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } } peers { permanent_uuid: "64394c7a27e34d098530337bd2e35d20" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } }
I20260924 00:25:13.746146 21942 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb0b1db8e1fd4acf87cf853c38c2da3e, State: Initialized, Role: FOLLOWER
I20260924 00:25:13.746279 21942 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } peers { permanent_uuid: "455b792d08a74a29bfd757771f3f29a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } } peers { permanent_uuid: "64394c7a27e34d098530337bd2e35d20" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } }
I20260924 00:25:13.746656 21942 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 00:25:13.746809 21916 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 00:25:13.746903 21916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:25:13.747848 21916 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } peers { permanent_uuid: "455b792d08a74a29bfd757771f3f29a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } } peers { permanent_uuid: "64394c7a27e34d098530337bd2e35d20" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } }
I20260924 00:25:13.748155 21916 leader_election.cc:288] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e [CANDIDATE]: Term 1 election: Requested vote from peers 455b792d08a74a29bfd757771f3f29a2 (127.0.98.61:35883), 64394c7a27e34d098530337bd2e35d20 (127.0.98.60:44325)
I20260924 00:25:13.748620 22066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "64394c7a27e34d098530337bd2e35d20"
I20260924 00:25:13.748750 22066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64394c7a27e34d098530337bd2e35d20 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:25:13.748852 21991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "455b792d08a74a29bfd757771f3f29a2"
I20260924 00:25:13.749032 21991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 455b792d08a74a29bfd757771f3f29a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 00:25:13.749482 22126 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } peers { permanent_uuid: "455b792d08a74a29bfd757771f3f29a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } } peers { permanent_uuid: "64394c7a27e34d098530337bd2e35d20" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } } }
I20260924 00:25:13.749612 22126 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:25:13.749686 22066 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 64394c7a27e34d098530337bd2e35d20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb0b1db8e1fd4acf87cf853c38c2da3e in term 1.
I20260924 00:25:13.749995 21891 leader_election.cc:302] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64394c7a27e34d098530337bd2e35d20, bb0b1db8e1fd4acf87cf853c38c2da3e; no voters:
I20260924 00:25:13.750200 21991 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 455b792d08a74a29bfd757771f3f29a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb0b1db8e1fd4acf87cf853c38c2da3e in term 1.
I20260924 00:25:13.750406 22126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 00:25:13.750804 22126 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e [term 1 LEADER]: Becoming Leader. State: Replica: bb0b1db8e1fd4acf87cf853c38c2da3e, State: Running, Role: LEADER
I20260924 00:25:13.750911 22126 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } peers { permanent_uuid: "455b792d08a74a29bfd757771f3f29a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } } peers { permanent_uuid: "64394c7a27e34d098530337bd2e35d20" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } }
I20260924 00:25:13.751448 22126 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb0b1db8e1fd4acf87cf853c38c2da3e. Latest consensus state: current_term: 1 leader_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } peers { permanent_uuid: "455b792d08a74a29bfd757771f3f29a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } } peers { permanent_uuid: "64394c7a27e34d098530337bd2e35d20" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } } }
I20260924 00:25:13.751549 22126 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e [sys.catalog]: This master's current role is: LEADER
W20260924 00:25:13.754870 22137 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 00:25:13.754941 22137 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 00:25:13.762298 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.750122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40984 (local address 127.0.98.62:44387)
0924 00:25:13.750205 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.750207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.751428 (+ 1221us) server_negotiation.cc:407] Connection header received
0924 00:25:13.751543 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.751545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.751557 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.751592 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.752583 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.753405 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.754600 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.754940 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.754984 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.758883 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:13.758885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:13.758885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:13.758894 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:13.760884 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.761341 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.761342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.761342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.761401 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.761614 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.761615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.761615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.761751 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:13.761856 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:13.762101 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 00:25:13.762190 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 00:25:13.763281 22147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 00:25:13.763510 22147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 00:25:13.768385 21991 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 455b792d08a74a29bfd757771f3f29a2 [term 1 FOLLOWER]: Refusing update from remote peer bb0b1db8e1fd4acf87cf853c38c2da3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:25:13.768115 22066 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 64394c7a27e34d098530337bd2e35d20 [term 1 FOLLOWER]: Refusing update from remote peer bb0b1db8e1fd4acf87cf853c38c2da3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 00:25:13.768774 22144 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "455b792d08a74a29bfd757771f3f29a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:25:13.768986 22126 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "64394c7a27e34d098530337bd2e35d20" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 00:25:13.769248 22021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.750488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42072 (local address 127.0.98.61:35883)
0924 00:25:13.750534 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.750536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.755066 (+ 4530us) server_negotiation.cc:407] Connection header received
0924 00:25:13.755250 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.755251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.755264 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.755302 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.756951 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.757754 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.763272 (+ 5518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.763629 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.763677 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.765675 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:13.765677 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:13.765677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:13.765685 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:13.767589 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.768134 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.768135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.768136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.768223 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.768508 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.768509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.768509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.768659 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:13.768762 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:13.769063 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 00:25:13.769150 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 00:25:13.770599 22112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 455b792d08a74a29bfd757771f3f29a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb0b1db8e1fd4acf87cf853c38c2da3e. Latest consensus state: current_term: 1 leader_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } peers { permanent_uuid: "455b792d08a74a29bfd757771f3f29a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } } peers { permanent_uuid: "64394c7a27e34d098530337bd2e35d20" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } } }
I20260924 00:25:13.770718 22112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 455b792d08a74a29bfd757771f3f29a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:25:13.771656 22126 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } peers { permanent_uuid: "455b792d08a74a29bfd757771f3f29a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } } peers { permanent_uuid: "64394c7a27e34d098530337bd2e35d20" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } } }
I20260924 00:25:13.771749 22126 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e [sys.catalog]: This master's current role is: LEADER
I20260924 00:25:13.772437 22149 mvcc.cc:205] Tried to move back new op lower bound from 7332698168384729088 to 7332698168324624384. Current Snapshot: MvccSnapshot[applied={T|T < 7332698168384729088}]
I20260924 00:25:13.772783 22098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64394c7a27e34d098530337bd2e35d20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb0b1db8e1fd4acf87cf853c38c2da3e. Latest consensus state: current_term: 1 leader_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } peers { permanent_uuid: "455b792d08a74a29bfd757771f3f29a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } } peers { permanent_uuid: "64394c7a27e34d098530337bd2e35d20" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } } }
I20260924 00:25:13.772913 22098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64394c7a27e34d098530337bd2e35d20 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:25:13.773028 22112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 455b792d08a74a29bfd757771f3f29a2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } peers { permanent_uuid: "455b792d08a74a29bfd757771f3f29a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } } peers { permanent_uuid: "64394c7a27e34d098530337bd2e35d20" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } } }
I20260924 00:25:13.773124 22112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 455b792d08a74a29bfd757771f3f29a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:25:13.773650 22144 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } peers { permanent_uuid: "455b792d08a74a29bfd757771f3f29a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } } peers { permanent_uuid: "64394c7a27e34d098530337bd2e35d20" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } } }
I20260924 00:25:13.773754 22144 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e [sys.catalog]: This master's current role is: LEADER
I20260924 00:25:13.774159 22098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64394c7a27e34d098530337bd2e35d20 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb0b1db8e1fd4acf87cf853c38c2da3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44387 } } peers { permanent_uuid: "455b792d08a74a29bfd757771f3f29a2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35883 } } peers { permanent_uuid: "64394c7a27e34d098530337bd2e35d20" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44325 } } }
I20260924 00:25:13.774276 22098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64394c7a27e34d098530337bd2e35d20 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 00:25:13.776783 22147 catalog_manager.cc:1347] Generated new cluster ID: ead62fbef4b44d47ae9aed0d409b6557
I20260924 00:25:13.776480 22150 mvcc.cc:205] Tried to move back new op lower bound from 7332698168384729088 to 7332698168324624384. Current Snapshot: MvccSnapshot[applied={T|T < 7332698168384729088}]
I20260924 00:25:13.777714 22147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 00:25:13.779590 22096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:13.751145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59032 (local address 127.0.98.60:44325)
0924 00:25:13.751208 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.751211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.764466 (+ 13255us) server_negotiation.cc:407] Connection header received
0924 00:25:13.764606 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.764607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.764621 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.764659 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.765517 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.766281 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.767432 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:13.767756 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:13.767820 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:13.775557 (+ 7737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 00:25:13.775559 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 00:25:13.775560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 00:25:13.775569 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 00:25:13.777601 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.778172 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.778173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.778174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.778262 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 00:25:13.778530 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 00:25:13.778531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 00:25:13.778531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 00:25:13.778688 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 00:25:13.778815 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:13.779385 (+ 570us) server_negotiation.cc:299] Negotiation successful
0924 00:25:13.779477 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 00:25:13.780779 22153 mvcc.cc:205] Tried to move back new op lower bound from 7332698168384729088 to 7332698168324624384. Current Snapshot: MvccSnapshot[applied={T|T < 7332698168384729088}]
I20260924 00:25:13.788761 22147 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 00:25:13.789633 22147 catalog_manager.cc:1504] Loading token signing keys...
I20260924 00:25:13.798429 22147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb0b1db8e1fd4acf87cf853c38c2da3e: Generated new TSK 0
I20260924 00:25:13.798741 22147 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 00:25:13.830066 21945 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:13.828110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40994 (local address 127.0.98.62:44387)
0924 00:25:13.828160 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.828163 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.829254 (+ 1091us) server_negotiation.cc:407] Connection header received
0924 00:25:13.829377 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.829379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.829392 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.829426 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.829985 (+ 559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40994: BlockingRecv error: recv got EOF from 127.0.0.1:40994 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 00:25:13.831035 22021 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:13.828610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42080 (local address 127.0.98.61:35883)
0924 00:25:13.828656 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.828658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.830432 (+ 1774us) server_negotiation.cc:407] Connection header received
0924 00:25:13.830546 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.830547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.830560 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.830594 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.830964 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42080: BlockingRecv error: recv got EOF from 127.0.0.1:42080 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 00:25:13.831952 22096 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:13.828937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59042 (local address 127.0.98.60:44325)
0924 00:25:13.828980 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.828982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.831359 (+ 2377us) server_negotiation.cc:407] Connection header received
0924 00:25:13.831472 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.831473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.831486 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.831519 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.831880 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59042: BlockingRecv error: recv got EOF from 127.0.0.1:59042 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 00:25:13.834883 22021 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:13.833942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42082 (local address 127.0.98.61:35883)
0924 00:25:13.833986 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.833988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.834254 (+ 266us) server_negotiation.cc:407] Connection header received
0924 00:25:13.834377 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.834378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.834399 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.834434 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.834827 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42082: BlockingRecv error: recv got EOF from 127.0.0.1:42082 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 00:25:13.838301 22021 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:13.837401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.0.98.61:35883)
0924 00:25:13.837440 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.837442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.837707 (+ 265us) server_negotiation.cc:407] Connection header received
0924 00:25:13.837835 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.837836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.837849 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.837882 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.838244 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42086: BlockingRecv error: recv got EOF from 127.0.0.1:42086 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 00:25:13.842743 22021 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:13.841766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42096 (local address 127.0.98.61:35883)
0924 00:25:13.841820 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.841822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.842117 (+ 295us) server_negotiation.cc:407] Connection header received
0924 00:25:13.842231 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.842232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.842244 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.842277 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.842688 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42096: BlockingRecv error: recv got EOF from 127.0.0.1:42096 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 00:25:13.849306 22021 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:13.848354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42104 (local address 127.0.98.61:35883)
0924 00:25:13.848398 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.848400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.848670 (+ 270us) server_negotiation.cc:407] Connection header received
0924 00:25:13.848788 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.848789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.848816 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.848851 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.849250 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42104: BlockingRecv error: recv got EOF from 127.0.0.1:42104 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 00:25:13.859867 22021 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:13.858835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42110 (local address 127.0.98.61:35883)
0924 00:25:13.858886 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.858888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.859176 (+ 288us) server_negotiation.cc:407] Connection header received
0924 00:25:13.859294 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.859295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.859308 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.859344 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.859793 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42110: BlockingRecv error: recv got EOF from 127.0.0.1:42110 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 00:25:13.878573 22021 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:13.877511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42118 (local address 127.0.98.61:35883)
0924 00:25:13.877560 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.877563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.877897 (+ 334us) server_negotiation.cc:407] Connection header received
0924 00:25:13.878019 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.878021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.878033 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.878067 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.878505 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42118: BlockingRecv error: recv got EOF from 127.0.0.1:42118 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 00:25:13.913185 22021 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:13.912228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42126 (local address 127.0.98.61:35883)
0924 00:25:13.912301 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.912303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.912553 (+ 250us) server_negotiation.cc:407] Connection header received
0924 00:25:13.912655 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.912656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.912667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.912721 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.913103 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42126: BlockingRecv error: recv got EOF from 127.0.0.1:42126 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 00:25:13.979640 22021 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:13.978722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42138 (local address 127.0.98.61:35883)
0924 00:25:13.978810 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:13.978813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:13.978904 (+ 91us) server_negotiation.cc:407] Connection header received
0924 00:25:13.979015 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:13.979016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:13.979031 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:13.979117 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:13.979565 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42138: BlockingRecv error: recv got EOF from 127.0.0.1:42138 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 00:25:14.109874 22021 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:14.109072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42154 (local address 127.0.98.61:35883)
0924 00:25:14.109264 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:14.109266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:14.109278 (+ 12us) server_negotiation.cc:407] Connection header received
0924 00:25:14.109306 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:14.109307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:14.109318 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:14.109390 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:14.109779 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42154: BlockingRecv error: recv got EOF from 127.0.0.1:42154 (error 108)
Metrics: {"server-negotiator.queue_time_us":151}
W20260924 00:25:14.367733 22021 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 00:25:14.367034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42168 (local address 127.0.98.61:35883)
0924 00:25:14.367111 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:14.367113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:14.367124 (+ 11us) server_negotiation.cc:407] Connection header received
0924 00:25:14.367183 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:14.367184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:14.367196 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:14.367283 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 00:25:14.367647 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42168: BlockingRecv error: recv got EOF from 127.0.0.1:42168 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 00:25:14.676786 22109 catalog_manager.cc:1259] Loaded cluster ID: ead62fbef4b44d47ae9aed0d409b6557
I20260924 00:25:14.676890 22109 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 64394c7a27e34d098530337bd2e35d20: loading cluster ID for follower catalog manager: success
I20260924 00:25:14.678082 22109 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 64394c7a27e34d098530337bd2e35d20: acquiring CA information for follower catalog manager: success
I20260924 00:25:14.678465 22109 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 64394c7a27e34d098530337bd2e35d20: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 00:25:14.703776 22123 catalog_manager.cc:1259] Loaded cluster ID: ead62fbef4b44d47ae9aed0d409b6557
I20260924 00:25:14.704128 22123 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 455b792d08a74a29bfd757771f3f29a2: loading cluster ID for follower catalog manager: success
I20260924 00:25:14.705042 22123 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 455b792d08a74a29bfd757771f3f29a2: acquiring CA information for follower catalog manager: success
I20260924 00:25:14.705415 22123 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 455b792d08a74a29bfd757771f3f29a2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 00:25:14.884222 22222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:14.881331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42178 (local address 127.0.98.61:35883)
0924 00:25:14.881455 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:14.881457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:14.881469 (+ 12us) server_negotiation.cc:407] Connection header received
0924 00:25:14.881528 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:14.881529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:14.881590 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:14.881686 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:25:14.881862 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:14.882412 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:14.883281 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:14.883514 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:14.883608 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:14.883796 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:25:14.883923 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:25:14.884035 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:14.884060 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 00:25:14.884088 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":53,"threads_started":1}
I20260924 00:25:14.888607 22228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:14.886162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59046 (local address 127.0.98.60:44325)
0924 00:25:14.886276 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:14.886279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:14.886306 (+ 27us) server_negotiation.cc:407] Connection header received
0924 00:25:14.886371 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:14.886372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:14.886406 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:14.886476 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:25:14.886567 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:14.887052 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:14.887845 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:14.888031 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:14.888086 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:14.888263 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:25:14.888380 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:25:14.888448 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:14.888527 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 00:25:14.888544 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
I20260924 00:25:14.893203 22234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:14.890397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41008 (local address 127.0.98.62:44387)
0924 00:25:14.890516 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:14.890519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:14.890529 (+ 10us) server_negotiation.cc:407] Connection header received
0924 00:25:14.890575 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:14.890576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:14.890647 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:14.890738 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:25:14.890950 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:14.891478 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:14.892243 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:14.892516 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:14.892627 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:14.892832 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:25:14.892919 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:25:14.893006 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:14.893068 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 00:25:14.893089 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":75,"threads_started":1}
I20260924 00:25:14.894024 22222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:14.890544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.0.98.61:35883)
0924 00:25:14.890599 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:14.890602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:14.890736 (+ 134us) server_negotiation.cc:407] Connection header received
0924 00:25:14.890847 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:14.890848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:14.890857 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:14.890886 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:25:14.891070 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:14.891913 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:14.893219 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:14.893486 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:14.893577 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:14.893713 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:25:14.893751 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:25:14.893805 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:14.893909 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 00:25:14.893939 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 00:25:14.894238 22228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 00:25:14.890841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59050 (local address 127.0.98.60:44325)
0924 00:25:14.890884 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 00:25:14.890886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 00:25:14.890925 (+ 39us) server_negotiation.cc:407] Connection header received
0924 00:25:14.891008 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 00:25:14.891009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 00:25:14.891019 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 00:25:14.891070 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 00:25:14.891256 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:14.892036 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:14.893302 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 00:25:14.893618 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 00:25:14.893664 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 00:25:14.893920 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 00:25:14.893965 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 00:25:14.894001 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 00:25:14.894125 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 00:25:14.894153 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 00:25:14.899865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 21874
I20260924 00:25:14.905460 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 21949
I20260924 00:25:14.910912 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYEWBIr/build/release/bin/kudu with pid 22024
2026-09-24T00:25:14Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2991 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5805 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (360599 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
2 FAILED TESTS